Overview the Arduino Sketch Uploading Process and ISP

FTDI is actually a brand of chips. The FTDI company specializes in chips used to connect via USB. In Arduino land, USB-to-serial chips are used to interface a computer running the Arduino IDE to your Arduino’s main processor for uploading new sketches and for interacting with your sketches via a serial monitor window. The USB-to-serial chip interfaces with the UART interface of the ATMEL processor on your Arduino.

In order for the Arduino to accept a sketch from the serial interface, it runs a program called a bootloader, which accepts the sketch and writes it into Flash memory. The bootloader actually resides in a small portion of the Flash memory in the upper range of memory addresses, which is reserved for bootloader use. As the bootloader program receives the sketch, it stores into the lower portion of Flash memory.

This serial programming uses a protocol called TTL serial. It is based on an old communication protocol called RS-232. RS-232 communications use voltages that change rapidly from a positive voltage of 3 to 25 volts, to a negative voltage of -3 to -25 volts. Personal computers typically no longer come with RS-232 interfaces, but they used to a few years ago. A PC with such an interface typically uses voltages that swing between 12 to -12 volts to send an RS-232 signal. The ATMEL processor on an Arduino uses signals in the 0 to 5 volt range. The version of this RS-232 protocol that is compatible with voltages in the range the Arduino processor can handle is called TTL serial, or sometimes you may hear it called TTL-232, or just serial.

Most Arduinos have a USB-to-serial adapter chip built-in, so you can connect the Arduino directly to your computer without any special interface or programmer. Some Arduinos, such as the LilyPad, Mini, and Pro Mini, don’t come with a USB-to-serial adapter and you need to supply your own external one. The external adapters are very often called FTDI adapters, even if the brand of chip is not FTDI.

There are several wires or connections involved with serial communication. The most important ones are called RX (short for receive) and TX (short for transmit).

As the name implies, the RX wire or pin receives from another device. It listens for incoming communication. The TX wire or pin transmits data to another device. The TX wire from your USB-to-serial adapter rapidly switches between 0 and 5 volts in a pattern, and it is attached to the RX pin on the Arduino’s processor, which is listening to those patterns.

The communication is two-way, so the Arduino’s processor can send signals back to the USB-to-serial adapter. The processor uses it’s own TX pin to send a signal to the USB-to-serial adapter’s RX pin. The USB-to-serial adapter translates the signals back and forth between the Arduino and your computer through the USB cable.

The computer sends the sketch to your Arduino, and receives the sketch back from the Arduino so the computer can verify that the program loaded okay. The same serial connection can be used by your sketch running on the Arduino to communicate back to you via the serial monitor window. It is common to use this method to debug your sketches, because you can sprinkle Serial.print statements in your code in strategic places to report back the value of variables or to tell what your code is doing at the moment. And you can send data back to your computer, such as sensor or input pin readings. You can also send data from the computer to the sketch running on your Arduino using the serial monitor.

You may wonder how the Arduino knows when serial communication is trying to send it a sketch to load, or just communicating through the serial monitor with the sketch that is already running on the Arduino? The answer is simple. The bootloader program that accepts sketches and reprograms the Arduino only runs in the first couple of seconds after the Arduino’s processor is reset. If the bootloader program running on the Arduino does not receive a certain sequence of characters indicating a sketch is attempting to upload, it quits running and starts the sketch that you loaded previously.

When Arduinos were first developed, the person uploading a sketch to the Arduino had to press the reset button on the Arduino right at the beginning of the upload process, and time it correctly, or try again. Later versions of Arduino made use of an additional serial protocol wire called DTR to reset the Arduino automatically. Since the DTR signal goes from 5V to 0 at the moment a new connection to the Arduino is started, if you send this signal to the reset line of the Arduino’s processor, it resets. If reason the new serial connection is being established is because the Arduino IDE is sending a sketch to the bootloader, the bootloader goes ahead and accepts the sketch and stores it. If reason the new serial connection is being established is because you’re opening the serial monitor of the Arduino IDE, then the Arduino resets, the bootloader runs just long enough to realize the IDE is not attempting to send a sketch, and the sketch you previously uploaded starts running.

It is useful to know about these signals such as the DTR signal used for automatic reset, especially if you want to correctly connect an USB-to-serial adapter to an Arduino that does not have the adapter built in, such as LilyPad, Mini, or Pro Mini. Or, if you want to troubleshoot problems with an Arduino, or program one Arduino using another. I mentioned in the previous paragraph the DTR signal goes from 5V to 0 at the moment a new connection is established. Well, this signal goes low and stays low for the entire time it is connected. If that signal were attached directly to the processor’s reset pin, the processor would stay in a state of reset and would never get started running the bootloader or the sketch you uploaded previously.

So, here’s the trick: The DTR signal is sent through a small capacitor to the reset pin of the processor. The capacitor turns the voltage from the DTR signal (which goes low and stays low) into a temporary spike that goes low to 0V and comes right back up to 5V. This resets the Arduino and lets it start the bootloader, which later starts the sketch.

A USB-to-serial adapter requires a driver to be loaded on your computer, so your computer will understand how to communicate with it. Genuine Arduinos use specific and finite list of models of USB-to-serial adapters so the IDE can be pre-packaged with all the drivers you may need. If you are using a clone, fake, derivative, home-made, or otherwise non-supported development board made by somebody other than arduino.cc or one of their partner manufacturers, that is actually just fine because it is open source hardware and software. Anybody can make one. But the board may use a USB-to-serial adapter that requires a driver for your computer and that is not included in the software download for the Arduino IDE. An example of this is the CH340G USB-to-serial chip that is very common on Arduino-compatible boards sold on eBay and Amazon. It is good for you to have knowledge of this so you prepare your computer with the correct driver.

This serial communication happening with the Arduino’s processor usually happens through specialized pins on the processor that are specifically designed for TTL serial communication. The processor has special hardware called a UART, which handles sending the data out of the TX pin or receiving data on the RX pin. The logic for handling these signals is built-in to the processor at a very fundamental level. If you are not using the RX and TX pins on the processor for serial communication using the UART, the pins are available for doing other things, such as activating relays, transistors, LEDs, or receiving digital signals from sensors. They are just plain digital pins if not occupied with serial signals. Some processor chips have multiple UARTs and therefore several sets of RX and TX pins. Those pins are also plain digital pins if not being used by the UART part of the processor for serial communication.

A bootloader can be designed to accept sketches via any protocol using almost any pins on the processor. The processor has a variety of pins that are plain digital pins but those pins can optionally be used for specialized protocols. For example, there is a group of pins that can be used for IIC or I2C protocol, which is a special serial bus protocol capable of communicating with multiple devices. And there is another group of pins that can be used for the SPI protocol, which is yet another different serial bus protocol used to communicate with devices. The processor has special hardware designed to implement these different protocols on the various dedicated pins, and the logic for handling these signals is built-in to the processor at a very fundamental level.

Twister #1: These various protocols such as TTL serial, I2C, and SPI have the low-level logic built into the processor for handling the protocols if the specific pins for the protocol are connected to another device, but the protocols can be implemented to work on any of the digital pins on the processor by writing software that emulates the behavior of the processor’s built-in logic circuits. This is called bit banging. A program can be written that drives the pins in a pattern that meets the requirements or specifications of the protocol. For example, a software serial protocol can be written that uses non-UART pins of the processor. The software then takes the place of hardware UART circuitry. This is useful if the pins you would normally use for a particular protocol are otherwise occupied, and you need to use other pins. Or, if you want to have more than the one built-in serial interface. For example, the ATmega328P processor has only one UART and it uses digital pins 0 and 1. It normally only communicates with one other serial device at a time. If you want to attach another serial device on pins 2 and 3, you can do that with a software serial program that bit-bangs the protocol. The Optiboot bootloader can be compiled to communicate through software serial on non-UART pins.

Arduinos commonly come with a serial bootloader installed, which uses the hardware UART onboard the ATMEL processor, and the Arduino IDE running on your computer is designed to communicate with that bootloader.

Twister #2: You don’t need a bootloader! Read about ISP next.