While you have your windows out don't forget to take a look at the window glazing gasket. It's not just dirt, most of the time that's mold and mildew that can be hazardous to your health! After you have spruced up the seals, use high-quality Weatherstrip adhesive which makes the difference between a clean, 5 minute job and a huge mess.
NEED TO CLEAN THE METAL AS WELL?
Use a fine grit sanding block. Make sure you run back and forth with the grain of the metal even as you go around the curve of the window. Note: this is just about the only time you should ever use sand paper on your Airstream!!! This aluminum is extruded and not the same as the Alclad sheeting on the exterior of your trailer.
Because you are working with fine particles of metal from sanding and removing mold spores, we always recommend protecting yourself properly. Do this in a well ventilated area. Wear gloves and most importantly a face mask to help keep your lungs and hands in good shape. Also, remember to take your time. Once you hit Airstream eras with curved windows or double panes, the price to replace them raises. Tackle this process without rushing to get the best results and to help avoid accidental drops and dings. You got this.