Two things effect the RATE of weathering.
1.) The composition of the material
2.) The texture of the material

Composition:
Quartz is very weathering resistant from the composition point of view. It has nearly no solubility in water.

Rock that contain high quartz content are thus very resistant to chemical weathering.

Texture:
If a rock is porous or has a granular texture, it can be weathered faster than one that is more continuous of phase.

Texture provides advantage to both chemical and physical weathering. Water's dissolution power is increased if it can cover more surface area. Water can also cause more physical weathering with the freeze/thaw cycle with more penetration into a rock.

Since chemical weather is a chemical reaction, those things that effect chemical reactions effect the rate at which something weathers. Most chemical reactions are increased when heat is applied and reduce in speed when cooled. Many chemical reactions run faster if the products are removed as the reaction continues. (Kinetics)

Hence weathering is known to be faster in hot, wet areas and slower in cold, dry areas. (Climate)

If the reaction happens on rocks that are exposed on a vertical surface, it may be accelerated because the products can easily fall away exposing more new material. Weathering on a flat surface tends to be slower. (Geometry)

Ants and other wild life can move vast amounts of grain or small rocks. They themselves do not cause weathering but may effect it.

Glaciers and Glaciation
Although glaciers act like great chains of sandpaper removing much of the rock they contact, their often have an opposite effect to weathering. If they move through granite or other weathering resistant rocks, they may actually polish the surface of the rock and make it more resistant to the weathering process.

Rate of Weathering
Weathering occurs more rapidly when it first begins, and then slows the longer it continues. There are a couple of general reasons for this.

As weathering must penetrate deeper and deeper in to a rock for it to react with new material, the time of penetration lengthens. The mass migration of dissolved products must also transverse more area and the fluids may be more nearly saturated.

Since chemical processes are based on steady state kinetics, the longer a reaction progresses the more likely it reaches a steady continuous state. At first it tends to run faster as there are more reactants present, but it will tend to slow as the concentration ratio of reactants to final products increase.