I am using LVM2 with linux 2.6.11. One of the drive in the volume is going bad. I could hear screeching noise while the platters were spinning.
That is bad. Not a problem, though, as I can just vacate the data from that drive using pvmove.
The bad drive is /dev/bad. /dev/avail is a volume with some free space.
VG Name vg0
PV Size 148.09 GB / not usable 0
Allocatable yes
PE Size (KByte) 4096
Total PE 37911
Free PE 25111
Allocated PE 12800
PV UUID e5EvaO-0oo5-Zenl-KzkY-wFf7-EkQX-di1mXt
--- Physical volume ---
PV Name /dev/avail
VG Name vg0
PV Size 186.30 GB / not usable 0
Allocatable yes
PE Size (KByte) 4096
Total PE 47694
Free PE 20658
Allocated PE 27036
PV UUID HLtCMK-751U-IFAW-5Aj3-FQ3w-5tqO-8svvwt
Let's move it to the only drive with available space, /dev/avail. /dev/bad has 12800 allocated PE while /dev/avail has 20658 free PE. I was not expecting any problem fitting the data in /dev/bad into /dev/avail.
# pvmove -i 5 -v /dev/bad
Finding volume group "vg0"
Archiving volume group "vg0" metadata.
Creating logical volume pvmove0
Moving 0 extents of logical volume vg0/lv0
Insufficient contiguous allocatable extents (1777) for logical
volume pvmove0: 12800 required
Unable to allocate temporary LV for pvmove.
Urk? It needs to be contiguous? What to do now?
Searching through the lvm mailing list shows that pvmove is dumb. It only sees the first free PE (physical extent). OK, let's work around this.
pv0 is the physical volume corresponding to /dev/bad. From this, we see that there are three segments residing in pv0. The first starts at PE 30720.
Let's try to fill that 1777 free PE on the dest drive, /dev/avail. That means, we'll be moving PE 30720 to (30720+1777-1=32496) from pv0.
# pvmove -i 5 -v /dev/bad:30720-32496
Finding volume group "vg0"
Archiving volume group "vg0" metadata.
Creating logical volume pvmove0
Moving 0 extents of logical volume vg0/lv0
Moving 1777 extents of logical volume vg0/lv1
Moving 0 extents of logical volume vg0/lv2
Moving 0 extents of logical volume vg0/lv3
Found volume group "vg0"
Updating volume group metadata
Creating volume group backup "/etc/lvm/backup/vg0"
Found volume group "vg0"
Found volume group "vg0"
Loading vg0-pvmove0
Found volume group "vg0"
Loading vg0-lv1
Checking progress every 5 seconds
/dev/hdf2: Moved: 7.7%
/dev/hdf2: Moved: 14.4%
/dev/hdf2: Moved: 22.6%
/dev/hdf2: Moved: 29.7%
/dev/hdf2: Moved: 37.4%
/dev/hdf2: Moved: 44.6%
/dev/hdf2: Moved: 52.3%
/dev/hdf2: Moved: 60.0%
/dev/hdf2: Moved: 67.7%
/dev/hdf2: Moved: 75.4%
/dev/hdf2: Moved: 83.1%
/dev/hdf2: Moved: 90.3%
/dev/hdf2: Moved: 97.4%
/dev/hdf2: Moved: 100.0%
Found volume group "vg0"
Found volume group "vg0"
Found volume group "vg0"
Loading vg0-pvmove0
Found volume group "vg0"
Loading vg0-lv1
Found volume group "vg0"
Found volume group "vg0"
Removing temporary pvmove LV
Writing out final volume group after pvmove
Creating volume group backup "/etc/lvm/backup/vg0"
Finally, after repeating the above process for the remaining segments, /dev/bad, aka pv0, is free of data and is safe to take down.
# vgreduce vg0 /dev/bad
Toss it in the garbage bin.
(originally from http://microjet.ath.cx/WebWiki/pvmove%20problem.html)
When you do M-x foo in emacs, you are asking emacs to execute an interactive function named "foo". The function must be declared in a special way as follow:
The form on the second line, "(interactive)", tells emacs that this function is an interactive function.
But how does the emacs knows you put that form in that function? It could try to execute the function, but it may produce an unwanted side-effect (like executing "rm -rf /" if the condition is met).
Could you do a partial execution, e.g.: when a function is defined, execute only the first form and don't execute the rest? But that does not explain the following capability:
When you do M-x foo/2, emacs would first run the read-string function, prompt the user, get the answer from user, and then run the function foo/2. But if you call foo/2 non-interactively by calling it from another function like in call-foo/2, emacs will not prompt the user at all.
Well, that blows the hypotheses that when a function is defined, emacs executes only the first form. There could be no execution whatsoever since doing so may cause severe side-effect. What if the execute form in call-foo/3 is executed? Bad, bad, bad.
So, where is the magic?
The magic lies in macro
defun is actually a special form call. A special form is a form that may be implemented and/or executed differently. If a form is implemented in non-elisp language, it is a special form. If a form is not a function call form, it is a special form (ordinarily, '(something)' in elisp would execute the function 'something').
A special form basically can do anything to its body (macro), including scanning for the interactive form when it is called, just like what the special form defun does.
(originally from http://microjet.ath.cx/WebWiki/HowDoesEmacsKnowWhichFunctionsAreInteractive.html)
In Debian Woody and Sarge (previous stable and current stable trees as of today), the ruby packages are split up into many packages. This has caused problems to Debian users not having enough dpkg-foo skill, which are many.
The situation is corrected in Debian unstable, but it would be a while before the correction trickles down to testing, and even longer to stable.
Quick, how do you encode a space in a URI? As '+' or as '%20'?
The answer is the standard 'it depends'.
URI was originally specified by RFC 1738. At that time, they were still calling it URL. The specification was revised and renamed to URI in RFC 2396. Since the URI was formalised in RFC 1738 before the importance of supporting internationalisation was recognised, RFC 2396 clarifies that unless communicated otherwise, one could assume the URI to be in US-ASCII character set.
The RFC acknowledges a URI may be composed of many components. It uses
<first>/<second>;<third>?<fourth>
as an example of a partial URI that has four components. The "/", ";", "?" symbols are components separators and are defined by each component's schema. The above separator symbols are just for example purposes.
Most of section 2 of the RFC talks about encoding character data into URI. It is long and hard to read.
Wouldn't it be simpler if it is presented as bullet points? Anyway, the gist of section 2 is as follow:
The escape syntax is "%" hex hex, e.g.: "%20" for space for the US-ASCII character set.
The characters in delims class MUST be escaped.
The membership of the reserved character class is fluid and depends on the context. "/" could be reserved in one context, and not reserved in another. If you want to use a reserved character in your data, you'd have to escape it.
An implication of the above point is that the membership of the unreserved character class is also fluid. Reserved's losses are unreserved's gains. To continue the example with "/", the "/" character would be added into the unreserved class if "/" is not reserved in a particular context.
All unreserved characters must be escaped
All unreserved characters may be used as-is.
Unreserved characters may also be escaped. For example, "~" (a mark) may be escaped as "%7e".
That's the generic URI spec. It is quite liberal in allowing specific schema to override the definitions of reserved and unreserved characters.
This allowance is used by the W3C's application/x-www-form-urlencoded specification to specify a different way of encoding values. Specifically, W3C's specification specifies that the space character must be encoded as '+', and the original members of the reserved class must be escaped (effectively removes the fluidity of the reserved class definition).
So, to answer the question: how do you encode a space in URI, the answer would be: in a generic context, as "%20", and in dealing with web forms (CGI as well), "+".
Long winded answer.
(originally from http://microjet.ath.cx/WebWiki/EscapingURIForCgi.html)
Several days ago I tried installing Linux on my laptop, Acer TM-290. I have usually been using Debian on my other laptops, but this time, I was wondering about Ubuntu.
So I downloaded and installed Ubuntu (Hoare) on the laptop. As usual, I always put the swap partition on the innermost track. That means the first partition.
On the first boot, the BIOS complained:
Hard disk boot sector invalid
Press 'H' to retry Hard Disk, any other key for floppy
Having never used Ubuntu before, I was suspicious of its installation process. In any case, Hoare was not a 'stable' release, just a preview release at this time. So, I proceeded installing Debian on it.
Same error.
Then I installed Win2000 on it and there was no error.
I was so confused. I tried debugging the bootcode loader in the MBR and found nothing wrong.
Then today I noticed that Win2000 installation always set the bootable flag of the first partition. I was wondering if that would have any effect at all.
It did. That was the trigger. The BIOS wanted the first partition to have the bootable flag set regardless of whether it is actually a bootable partition. I always put the swap partition as the first partition. So, I now have a bootable swap partition. Whatever that means.
(originally from http://microjet.ath.cx/WebWiki/WhenBIOSBecomesStupid.html)
Recently I went for an interview where I was asked this question: How do you load thirty millions entries from a file into a database within 4 hours.
I was puzzled on its significance. I have been using postgresql and loading 30 million entries is nothing special at all
Turned out they were using MS SQL. I was not sure what the deal was with batch loading of 30 million entries in MS SQL as I have minimum experience with that product. They cited a problem they encountered: apparently, they could not load 30 million entries within 4 hours without resorting to a technique which they had expected me to answer. Also, they said there was a problem with transaction log being overflowed. Beat me up, but I don't expect MS SQL to be having such problems, especially not on a high-end database machine that they had (it had 24 CPU, and I assume an equally impressive disk system and memory)
I was not able to come up with an answer that they were looking for, that is, to break up the data into sections and bulk-load each section from separate session. Each session was to be initiated from a different computer, so as to reduce the load. (Note, though, that the question was highly MSSQL-specific. As I demonstrate below, postgresql has not any problem with loading 30 million entries).
Of course, the point of their question was not that loading 30 millions entries was hard, but rather, how do you load a large data within as short a time as possible.
So I was left thinking how loading from different sessions could hasten the loading process. Sure, I've read debates on postgresql mailing list on how loading from multiple sessions could reduce the time, but I have never paid any attention to it since I was not interested in it at that time.
I did not know much about the subject and had no opinion on that. So I accepted their answer for the time being. But, ever the curious type, I decided to conduct my own experiment.
I set on trying to answer three questions:
Is loading 30 millions data in 4 hours hard to accomplish with postgresql?
Does postgresql have any problem with loading 30 millions data in a single transaction?
Does loading data from multiple sessions hasten the process?
For questions #1 and #2, the experiment was done on my laptop, a 1.3 GHz Pentium M with 512 MB RAM and a 4200 RPM disk, and running the assorted programs that I usually run: emacs, X, kde, etc.
This was, by no means, a dedicated database nor powerful machine. The experiment was going on while I continued doing what I usually do: web browsing, software development, chatting, etc. I do not expect to get consistent results between runs due to other ongoing activities nor do consistent results matter much since what I was looking for was an upper bound.
The experiment to answer question #3 was performed on a two 3 GHz Pentium 4 machine with 512 MB RAM and a RAID-5 array with 10K RPM SCSI disks. This was a dedicated database machine and I expected to get a more consistent results.
I expected that if multiple sessions turn out to hasten the process, then the gain would depend on how many indexes in the table. Data loading itself is an IO-bound activity, so, even with multiple sessions, the total write throughput would still be constrained by the maximum write rate of the disk system. On the other hand, indexing would involve a significant amount of computation. So, that would be limited by the amount of computing power available. You could probably run more than n sessions on a n-CPU computer since the CPUs would be idle some of the time while waiting for disk I/O. Yet, at some point, the overhead of context switches would be too big, so certainly there is a limit on the number of sessions. What is the limit? I don't know but I suspect that would depend very much on your system.
Questions #1 and #2
First, I write a simple script to generate the data:
Sure, the data file they are importing would not be this simple. But that should not matter much since the most important thing, i.e.: reserving a row for the new data, is done equal amount of times. Populating the row mostly depends on your write throughput of your I/O data.
Then I generated 4 data files:
All the column names are fictitious.
I did not declare a primary key yet because I wanted to time data loading without any indexing as having a primary key implies having a unique index.
thirtymillion=# copy transactions from '/tmp/data.txt';
COPY
Time: 1639971.939 ms (27m20s)
So, 27m20s to load the first 30 millions. That's not bad at all. But how much time does indexing takes?
thirtymillion=> alter table transactions add primary key (tid);
NOTICE: ALTER TABLE / ADD PRIMARY KEY will create implicit index "transactions_pkey" for table "transactions"
ALTER TABLE
Time: 1374100.087 ms (22m54s)
The total for loading and indexing the first thirty millions was: 27m20s + 22m54s = 50m14s.
Good news! That was nowhere near 4 hours! Could this be a fluke?
Let's load the next three thirty-million datasets:
Don't heed the 0%CPU figure, as that's the CPU consumption of the client (psql), and not the server.
But, as you can see, the figures: 38m20s, 44m01s, and 45m03s are not that different from each other, and more importantly, all are under 4 hours. Notice that loading the first dataset took significantly longer than the rest. I do not know exactly why that was the case, but my hypothesis is the disk cache effectiveness is low if the loading and indexing are done separately. Perhaps the hypothesis is wrong, but what is important is I could now answer the first two questions:
Is loading 30 millions data in 4 hours hard to accomplish with postgresql? No
Does postgresql have any problem with loading 30 millions data in a single transaction? No
Question #3
I setup several data files, each contained 30 million entries for four sub-experiments:
Time separate loading and indexing as in the previous experiment
Time combined loading and indexing.
Time two concurrent loading and indexing sessions. The machine had two CPUs, so each CPU handled at most a session.
Time six concurrent loading and indexing sessions. The idea was to simulate high CPU and IO consumptions condition.
The time, 12m26s, about half of 27m20s as in the laptop case, was as expected since loading should be an IO-bound process. A 10K RPM disk should be able to write data about twice as quickly as a 4200 RPM disk.
Indexing it:
$ echo "alter table transactions add primary key (tid)" | /usr/bin/time psql thirtymillion
NOTICE: ALTER TABLE / ADD PRIMARY KEY will create implicit index "transactions_pkey" for table "transactions"
ALTER TABLE
0.00user 0.01system 8:55.03elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+2024minor)pagefaults 0swaps
Total time for the first batch was: 12m26s + 8m55s = 21m21s.
As expected, indexing benefited from a more powerful CPU. Then, could concurrent sessions reduce the total time simply because that uses the other CPU? Before I went on answering that question, I timed the combined loading and indexing of the data.
I didn't really pay attention to the CPU consumption during the run, so I couldn't tell if the slowdown was due to the process being CPU-bound or IO-bound.
But I now can answer question #3:
Does loading data from multiple sessions hasten the process? No.
If anything, loading data from too many sessions actually slows down the process.
By the end of the experimentation, I've loaded more than 500 million rows into pgsql at a remarkably steady rate of 30 million rows /15 mins. I am not sure why they needed 4 hours to do the same. Even accounting for various referential integrity checkings that may be present, 4 hours is still 16x 15 mins.
some emacs interfaces for external programs/scripts (like what xtla is to tla)
porting some project build procedures to elisp (all programmers here use emacs)
That's basically the extent of my emacs resume.
After writing enough of those interfaces, I find that it would actually be better in some (many?) circumstances if the program drives the emacs interface, instead of emacs driving the program, i.e: the user interface of the program is the emacs itself.
I am thinking of a emacs-as-user-interface library from where programs can control what is being displayed by emacs and also propagate user response back to the program.
With the library, the program says what to display, where that radio button goes, where that text field goes, and what code to invoke when an event occurs (callback).
The program will be a long-running one; programming to this library should be just like programming with any other UI toolkit: you setup the screen, then you go in a loop.
But why go to those trouble?
Integration, that's why.
That is also the same reason for the elisp code I've written (see above). I want the count-line functionality, spell checking, programmable completion, ability for other elisp code to interact (only on buffer-level).
All I have seen is emacs integrating with external programs; doesn't emacs have enough functionalities of its own to entice other programs to integrate with emacs?
(originally from http://microjet.ath.cx/WebWiki/EmacsAsUserInterface.html)
Last couple of weeks, we decided to switch from JDBC to Hibernate in a project. We were tired of the tediousness JDBC demands. So, after briefly evaluating Hibernate, we decided to make the jump.
Turns out there are many details that we did not think as important when we glossed over the Hibernate manuals.
The biggest problem is that the manual is more of a tutorial rather than a reference. We had difficulty doing things that are not covered in the manual, such as defining a 1-1 relationship: how to make sure that both entities share the same id. Another example: what does Session.lock(obj, Lock.UPGRADE) do to an entity without a version field (how can it know that it is performing an upgrade when refreshing from the database without the version information in a clustered environment).
We spent a lot of time figuring out the best way to adopt Hibernate into the system. There is even a time where we spent a whole week diagnosing Hibernate-related problem, then after that another week to refactor existing code in light of a better understanding on Hibernate.
The refactoring could go on for ages because as we understand Hibernate more, the urge to refactor/clean up becomes more prominent.
One day, we woke up and realised that we have been spending too much time tinkering with the system instead of building it. We decided to make a cut, a non-optimal cut, and proceeded to continue what the client paid us to do.
(originally from http://microjet.ath.cx/WebWiki/LosingOnesIntention.html)
Since the new manager move into my apartment, she has been a great study for me as an example of bad manager. Below is her first letter to the tenants. Emphasis by me. Capitalisation is as per the original letter.
August 14, 2002 From: Female PHM (pointy-haired manager)
Dear Residents,
The work is underway on our exterior improvements. Clubhouse should be done by Mid September. The exterior painting will begin the first week of September. We are also replacing and adding new exterior lights. The property will be illuminated. A few issues to bring up:
* Pet owners must begin cleaning up after their pets. I have SPIES in each building that are reporting back to the office who is walking their dogs with out picking up the mess. We are taking note of the apartment numbers of the pet owners. You will lose the privilege to have a pet and could be fined if you do not begin cleaning up the feces. It is a health hazard and unsightly. It is especially disgusting by building "B".
* WE are still have parking problems. Due to the lack of parking by building "A" THE ONLY AUTHORIZED residents to park in front of "A" building are "A" building residents. You will be towed without warning if you park in an assigned spot, or in front of "A" building if you don't live in "A" building. The other reserved numbered parking is $25.00 per month. If you wish to have assigned parking, stop by the office. With payment, we will issue a spot and you will receive a parking permit.
* There are no vehicle repairs allowed on the property. If you continue to work on your cars, you will receive a 20 day vacate notice for nuisance and waste.
* We will keep closing the pool if you don't stay out of it with street clothes. Your choice.
Sincerely,
PHM Property Manager
---
END
(originally from http://microjet.ath.cx/WebWiki/HowNotToBeAManager.html)
