2008-03-01
Brake pedal needs to be adjusted after master cylinder replacement
I consulted with my brother who was an actual certified mechanic. He suggested that that the brake pedal free play needed to be adjusted because of depth differences of the piston cap in master cylinders. At first I was skeptical. True, the replacement part was a remanufactured one; but was it really that different than the old one?
I measured and found that the freeplay distance was out of spec: 12.7 mm instead of 5 mm. But what could a 7.7 mm difference do to the braking performance?
After adjusting the freeplay distance, I found that I couldn't floor the pedal anymore. Although the wheels still won't lock up, the brake is now good enough for everyday usage. I attribute the inability of the brake to lock up the wheel to older (weaker) brake hose and moderate glazing on the front rotors and pads.
Replacing brake master cylinder
Apparently, the brake fluid had not been changed in a long time. After changing it, the brake pedal became soft, mushy and the pedal could go to the floor. The car became dangerous to drive. Even with pedal on the floor, the wheel would not lock up. Repeated bleeding didn't help.
Then I stumbled on an explanation somewhere in the Internet (I forgot to note the URL). The old brake fluid could have been saturated with water. It caused the rubber seal within master cylinder to swell. Because brake fluid (DOT3&4) is hygroscopic, when the fluid was changed it pruned the seal, leaving it wrinkly like your fingers when you have been swimming for too long.
This causes the master cylinder to have an internal leak: instead of forcing the fluid along the brake lines to the slave cylinders, the seal allows the fluid to leak past them to the unpressurised region.
Symptoms of a master cylinder with an internal leak:
- You can push the pedal to the floor with light continuous pressure even without the help of brake booster.
- The pedal slowly sinks to the floor under its own weight (by itself).
Since diagnosing brake failure cannot be done with 100% certainty, if you have one of the above symptoms and you cannot attribute it to anything else (leaky slave cylinders, leaky brake lines, weak brake hoses, various other kinds of external leak), you need to replace the brake master cylinder.
Special Tools & Parts:
- Replacement brake master cylinder. I got a rebuilt/remanufactured one for $60.
- Flare-Nut wrench (see picture to see the difference).
- Strong strings (optional. to help seat the reservoir in the new unit).
- Aluminium foil / plastic food wrap / plastic bag / any barrier suitable for containing brake fluid.
- Bench vise. Bench bleeding is much easier with one. Much much much easier.

Procedures:
- Lift car on all wheels with wheels off.
- You need some space around the brake master cylinder.
- So, start by clearing up the area surrounding it.
- Clean the area around master cylinder with brake part cleaner and wiping (physically). You want to get the area as clean as possible to prevent contaminant in your brake system. Use tooth brush (not the same one you used this morning) to clean tight spaces.
Do you know that brake part cleaner (Tetrachloroethylene) is a carcinogen? Do not let it drip to the floor. Do not let it free in the environment. I find an oil drain pan suitable to contain brake part cleaner liquid run-off. It's wide and short and can fit under various areas of the car. Don't use the same one as the one you use for engine oil. The chemical in the cleaner will render the oil unsuitable for recycling back as engine oil. It will force the recyclers to use it as fuel.
- With the master cylinder is still fastened to the brake booster, loosen the brake line nuts using flare-nut wrench. The reason you need to use flare-nut wrench is because they are made of soft material and tighten pretty tight (13-20N.m or 10-15 ft/lb). Don't open them, don't let fluid drip. Just loosen them and tighten loosely back with finger.

- Remove the master cylinder from the booster.
- Now that you can manoeuvre the master cylinder slightly, put aluminium foil or fluid barrier under and to the sides of it because next you'll be removing the brake lines. Put some towels (that's the white thing on top of the aluminium foil in the picture below) too so the caught fluid is not sloshing around on the barrier. When the brake lines are separated, they won't drip fluid, but the master cylinder will. So, put it in a plastic bag as you carry it over the car.
- Empty the master cylinder and reservoir of brake fluid.
- Mount the master cylinder on the bench vise.
- Remove the reservoir by moving it left and right while pulling it up. It's going be hard but it won't break as long as you are using your hand. The bench vise allows you to use both hands for pulling.
- Pry off the filter screen in the reservoir
- Clean the filter and the inside of the reservoir with brake part cleaner. Don't forget to do it above the drain pan to increase your karma. Afterwards, rinse with fresh brake fluid. Pour in some fluid into the reservoir, put on the cap and cover the ports with your fingers. Shake shake shake and let the fluid out. Repeat as necessary. Don't install the the filter yet, but clean it as well.
Here is a picture of the old and new master cylinders with the reservoir and metering valves removed.

- Install the reservoir, metering valves and the bench bleed plugs (the green plug screwed to the metering valves and ports in the picture below). If the plug does not fit well, do not use it because it won't help in eliminating air during bench bleeding. You need to be prepared for getting messy during the bench bleeding, or do a long painful re-bleed session on all wheels (good thing you already have the car up, right?)
Putting back the reservoir was hard for me as I was not powerful enough to press it in. So, I used a long piece of string to help convert torque to pressure. Make sure that the string form a wide band so the pressure will not damage the nylon reservoir.
- Install clear tubings from the plugs back to the reservoir. My green plugs takes 3/16" ID (internal diameter) tubings.
- Fill reservoir with fluid.
- Use a long screwdriver or a socket extension to push the piston repeatedly until you see no more air bubble. If there is a port without a plug (because it does not fit well), cover the port with your finger to prevent air being sucked in before releasing the pressure on the piston.
- Installation is the reverse of removal. Do not forget to put the master cylinder in a plastic bag during transport as it will be dripping fluid. Take your time in screwing in the brake line nuts. Do not cross-thread the master cylinder (the nut is soft, but the master cylinder is softer). Do not get panicky because the fluid is dripping. That's why you put the barrier underneath, right?
If you did a good bench bleeding, you don't have to do anything else beside putting the car down. Otherwise, go bleed the brake. Yes, new master cylinder means you can push the pedal to the floor without damaging the seal.
Make sure to test drive it and check for external leak.
Keep checking for five days for any fluid level drop in the reservoir and for external leak. Some leaks are very small. I found a very small leak on the third day on one of the the brake line nuts that was not there previously.
2008-02-22
Test, test, test
At some level, I have known this subconsciously. Want to have a successful Math exam? Do practice questions until you're bored to death. Having a 5-chapter History exam tomorrow? Do practice questions until dawn.
I think this also applies to things in general: expertise comes from repeated doings. In other words, expertise comes from experience.
2008-02-20
Disabling LKM in Debian GNU/Linux
You may want to do this on your public servers to help keep off LKM trojan.
apt-get install lcap
lcap -c CAP_SYS_MODULE
There are more tips at Taking Advantage of Linux Capabilities.
What is in a machine name?
machines are cheap. It is very cheap to have a specific (virtual)
machine for a specific purpose.
Already, I had several specific functionalities I wanted to have
implemented: intranet-wide (read: home) file server, intranet-wide
access to scanner, public-facing web server, public-facing mail
server, fresh OS base to test installation procedures on, etc.
Now they can each be implemented in its own machine. There will be
enough machines for me to justify entertaining some
system-administration practices, like impersonal machine naming.
I had always been naming my machines by the name of people I liked,
had crushes with, or dated. I cannot continue with that schema without
jeopardizing my marriage and lying to myself as I cannot start liking
someone just because I need to name a new machine.
A machine naming convention that I've always liked is one that encodes
the purpose and asset ID into the name. For example: WWW001 is a web
server machine, MAIL001 is a mail server machine, and WS001 is a
workstation machine. Some even go further by including physical
location information: DC06MAIL001 could be a mail server in data
center #6 in Washington DC.
My need for my simple system is simple. I choose the simple prefix "M"
which stands for machine. The names will be M01, M02, M03, and so on.
Reaching M99 is a strong indication that the system is no longer
simple and is a justification to use a more sophisticated naming
convention.
The machine name forms the basis for the canonical DNS name. Machines
with a single IP have similarly named DNS entries under the 'machine'
subdomain, e.g.: m01.machine.example.com.
Multi-Homed machines (having multiple IPs) also have similarly-named
DNS entries pointing to each of its interface. The entry names should
be descriptive enough to identify the interface, e.g.:
m01-internet.machine.example.com and m01-lan.machine.example.com if
m01 is an Internet-facing firewall machine.
Each interface may be assigned additional alias DNS name (CNAME
records). For example, if m01.machine.example.com is originally a
machine in LAN and is being upgraded to serve as an Internet-facing
firewall machine, then m01.machine.example.com then cease becoming
the canonical name. It may be dropped or added as an alias.
Publicly-Accessible servers also benefit from having an alias DNS
name, e.g.: www.example.com may be an alias for
m54.machine.example.com.
LKM in my server. Rootkitted.
From: root@example.com (Cron Daemon)
Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts
--report /etc/cron.daily )
To: root@example.com
Date: Sun, 27 Jan 2008 09:25:10 -0500
You have 1 process hidden for readdir command
You have 1 process hidden for ps command
chkproc: Warning: Possible LKM Trojan installed
When I received that email from one of my public servers, two
questions popped up: did I just get rootkited again?
I had been rootkitted about three years ago. At that time the only
network-accessible services on the computer was a stock apache 1.3
serving static pages and an unpatched qmail. I didn't have tripwire
setup and ended up reinstalling the system from scratch because I
didn't know which files had been compromised.
This time, I had tripwire (AIDE, actually) setup and was able to
identify the infected files. Yet I still ended up reinstalling the
system from scratch partly because things were too easy for me and
mostly because I just found the excuse I needed.
Firstly, the compromiser didn't take out the regular chkrootkit
report. Secondly, subsequent chkrootkit report didn't report any
warning. Did the chkrootkit itself was compromised? Later verification
with an uncompromised AIDE data file showed that it was not, but
still, you wouldn't know that then. Thirdly, the same question for the
AIDE binary itself. Fourthly, it was really painful trying to do an
off-line verification on a remote system. Fifthly, was it apache 1.3
again?
One thing I discovered was that Linux allows you to disable loadable
kernel module (LKM) which supposedly makes it impervious to LKM
Trojan, like the one I had. No use crying over spilt milk.
The server was a Xen virtual machine hosted by quantact.com. It was
setup two years ago. This break-in gave me an excuse to reorganize my
hosting choices. Around last year, they started offering a cheaper
hosting option on OpenVZ. So I tried it and found that I liked it. I
liked it so much I installed one at home.
OpenVZ does not allow virtual machines to load kernel modules. I hope
that is enough to circumvent the same attack vector.
New site
reorganise.
I had been thinking of using a public blog site, like blogger.com, for
my infrequent writings but had hesitated because of the perceived lack
of a decent editor to compose the entries with. I did not know about
the publish-via-email capability only until recently.
This capability allows me to keep using my favourite editor to create
a new email, do the compositions and send it to blogger.com for
publishing.
The ability to compose entries using the web interface is not
something to discount either. In my travelling, I have found hotels
with an over-eager filtering mechanism that restricted access to my
own web site because of its 'pornographic nature'. Yes, it is
pornographic only if you are turned on by boring technical writings.
2007-07-14
REST-ified Message Queue on HTTP
REST-ified Message Queue on HTTP[1]
In a post to rest-discuss mailing list, Paul Winkler noted that the REST API offered by activemq is not actually REST-ful at all.* it's not REST that says GET is a read-only operation; it's HTTP. So their HTTP implementation is broken. Sadly seems to be pretty common. * DELETE on a URL representing a queue means you want to delete the entire queue, not a single message!
He ended with a question "Has anybody thought of a good way to model a queue?". The following is my solution.
A message queue can be thought of as having two ends: one for the producers and the other for the consumers. The producers insert messages into the consumers which will be popped by the consumers. A queue is a simple concept and can be implemented simply in programs. However, once networking capability creeps into the requirements, things can get complicated because network is not reliable. Simple implementations of queue work only because it assumes data transfer within a process is reliable which is a reasonable assumption for in-process activity.
When network is factored in, new problems crop up:
How do you reliably insert messages?
This problem can be paraphrased as doing reliable delivery over HTTP. There are two techniques that address this: Bill de hÓra's HTTPLR and Paul Prescod's.Using any of these two technique, a producer can POST a message to the same queue URL shared by other producers.
==> POST /queue
some message
<== 201 Created
Location: /message/433How do you reliably allow multiple consumers to pop messages?
This problem expands on the previous one. There is a pop action that has to be considered. In a queue, the pop operation removes and return the head of the queue to the caller. With ActiveMQ, you'd send a DELETE to the queue URL. As Paul Winkler noted, that means the DELETE is applied to the queue, not to individual message.If one can get a list of messages, then one can repeatedly attempt to pop (by doing DELETE) on each messages. That is, one repeatedly poll for an available message.
==> GET /queue/messages
<== 200 OK
[list of messages]
// oops, some other client has taken msg #100
==> DELETE /message/100
<== 410 Gone
==> DELETE /message/102
<== 200 OK
[message 102]The polling technique is a form of busy waiting. It is an efficient technique if there is a high probability that the conditions will be met within a short period of time relative to the frequency of polling. To reduce collisions (the 410 Gone in the example above), consumers can try to DELETE randomly.
However, with an increase in number of clients, the frequency of collision increases and efficiency decreases. Furthermore, network is not reliable and client/server can die at any point. When a client issues a DELETE, and it does not receive any response back from the server, it cannot be certain that a message has not been popped for it.
Desirable Properties
So, what are the desirable properties a solution must have?Client should not Care About Other Client's Existence
The reason polling is done by clients in the example above is because a client has to care about other clients. If it knows for certain that it is the only client, it does not have to poll.We can put each client in its own virtual world where it can think that it is the only client in existence so it does not have to poll. This is accomplished by giving each client its own access point. Let's implement it by giving each client its own personalised de-queue process resource identified by a unique (per client) URL.
Forgiving to Unreliable Network, Client and Server
In real world, things break unexpectedly2, be it some network admin unplugging a port incidentally, or the server runs out of disk and errors out temporarily, or the client was killed by a OOM condition.Any implementation of a reliable message queue must be able to at least recover from the above breakages. We can simplify the possible conditions to two cases:
- Server does not receive the request
- Client does not receive the response
A Solution
The only thing required is for the consumer client to have its own consumer ID. This is not an unreasonable requirement and can be implemented easily (GUID or other simpler mechanisms).A de-queue is implemented as a sequence of one or more GET requests followed with a DELETE. A GET causes the server to returns a previously bound message to the consumer, if there is any. If there isn't any, return the next unbound message. A DELETE on a message removes the binding on that message.
==> GET /queue/pending_message?consumer_id=12345 <== 200, MSG #32 // repeated GET should yield the same message ==> GET /queue/pending_message?consumer_id=12345 <== 200, MSG #32 // A way to access a particular message ==> GET /queue/message/32 <== 200, MSG #32 // Unbind message 32 ==> DELETE /queue/message/32 <== 204 // Since no message is bound to the customer, this GET binds one GET /queue/pending_message?consumer_id=12345 <== 200, MSG #34 DELETE /queue/message/34 <== 204 // A personal access point allows personalised retrieval criteria POST /queue/pending_message/filter?consumer_id=12345 <==Return oldest message first <== 204 // Now retrieves from the oldest to the newest GET /queue/pending_message?consumer_id=12345 <== 200, MSG #1
[1] I explicitly mentioned 'on HTTP' because there are groups who seem to tied REST and HTTP too strongly for my liking; some even goes as far as equating REST to HTTP.
[2] Does expecting things to break unexpectedly make such breakage expected?
2007-05-17
Surrogate key does not make normalisation
However, it is not a magic dust and sprinkling it liberally over a database design does not automatically normalise the design. Such myth, unfortunately, has not gone away and seems to be spreading wider. It is as if for every person enlightened, three more replaces him/her.
I think the myth is propagated by the popularity of ORM and difficulties of many ORMs in dealing with relation having non-surrogate keys. But that's a topic for some other time.
Take, for example, the following design:
Surrogate Key
Food
| Food Id | Food |
|---|---|
| 1 | Apple |
| 2 | Orange |
| 3 | Yam |
| 4 | Carrot |
| Kind Id | Kind |
|---|---|
| 1 | Fruit |
| 2 | Root |
| Food Id | Kind Id |
|---|---|
| 1 | 1 |
| 2 | 1 |
| 3 | 2 |
| 4 | 2 |
Would you believe that it is equally as normalised as the following:
Natural Key
Food
| Food Id |
|---|
| Apple |
| Orange |
| Yam |
| Carrot |
| Kind Id |
|---|
| Fruit |
| Root |
| Food Id | Kind Id |
|---|---|
| Apple | Fruit |
| Orange | Fruit |
| Yam | Root |
| Carrot | Root |
and vice-versa.
The natural key version is not any more de-normalised nor redundant than the surrogate key version. There is no more redundancy in any one version than the other. One may be tempted to point at the
Food Kind relation of the Natural Key version and shout "AHA! There are two 'Fruit' and two 'Root' entries under the Kind Id column. That's redundant." I'd shout back "'AHA!' There are two '1' and two '2' entries at the same place in the Surrogate Key version. They both are keys. They uniquely identify tuples. If you are referring to the same tuple multiple times, there will be that many copy of the key be it represented with integers or strings.
This article is not debating which key is 'better'. There are many articles addressing the pros and cons of each key type. Simply google for "natural surrogate key".
There are other kind of having multiple copies of the same values. It is known as denormalised state.
De-Normalisation and redundancy goes hand-in-hand. A relation (table in SQL-speak) contains redundant information if the entries contain derivable information. The goal of normalisation is to perform loss-less decomposition such that there is no more derivable information in any relation.
De-Normalised Design
Food
| Food Id | Is Crunchy |
|---|---|
| Apple | true |
| Orange | false |
| Yam | false |
| Carrot | true |
| Kind Id |
|---|
| Fruit |
| Root |
| Food Id | Kind Id | Is Crunchy |
|---|---|---|
| Apple | Fruit | true |
| Orange | Fruit | false |
| Yam | Root | false |
| Carrot | Root | true |
The above is de-normalised design because the Is Crunchy field in Food Kind relation is derivable from Food itself.
Normalised Design
Food
| Food Id | Is Crunchy |
|---|---|
| Apple | true |
| Orange | false |
| Yam | false |
| Carrot | true |
| Kind Id |
|---|
| Fruit |
| Root |
| Food Id | Kind Id |
|---|---|
| Apple | Fruit |
| Orange | Fruit |
| Yam | Root |
| Carrot | Root |
2007-05-11
Hypermedia, the forgotten aspect of REST
Without looking at the slides and the following posts, can you guess what was missing?
If someone is going to give a presentation on REST, it is likely that they will emphasise the more popular aspects of REST: resource, resource identification, representation of the resource and the uniform methods for interacting and manipulating the resource.
Many presentations that I have seen would further concentrate on the look-and-feel of "REST-ful" URLs, giving the audiences the wrong message that REST is primarily about URL construction. Josh Sled brought a good example of how even a big-name (I'm not telling who, go figure it out from the look-and-feel of the URL) development group got REST wrong: http://developer.yahoo.com/photos/V3.0/createAlbum.html. Whatever do they mean by "Passed in as a REST parameter.", I wonder. Quick, the server ate too many REST parameters, pass in the non-REST ones.
What is missing is hypermedia. It's the last constraint mentioned in the dissertation. http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm section 5.1.5:
REST is defined by four interface constraints: identification of resources; manipulation of resources through representations; self-descriptive messages; and, hypermedia as the engine of application state.But whatever does "hypermedia as the engine of application state" mean? I like Mark Baker's explanation:
Hypermedia as the engine of application state" is simply a long-winded way of saying that REST clients make progress via links embedded in the data they retrieve. That sounds like an almost information-free statement, but it's not difficult to violate this constraint, and it can be quite costly to do so as well.For example, if you have a blog, and there is an entry for 2007-05-11, from your home page, there should be a link pointing to that entry. User and client should not be expected to guess how to get there.
Just as state is explicitly represented and transferred, so must the transitions.