Comic books didn’t really take off until the introduction of the Man of Steel in Action Comics 1 (Jun 1938). Before that, many comics were compilations of strips similar to the Sunday funnies, and some of these had time travel. The earliest series that I found was the story of Bobby and Binks, two kids who at first time traveled through a Magic Crystal of History and later just viewed past adventures through the crystal. They first appeared in DC’s first comic book publication, New Fun Comics #1, February 1935, by Adolphe Barreaux. As I find other such series, I’ll add them to my time-travel comic book page. So far, the pre-1939 titles I've found are:
Binks: Why—why—I can understand what they’re saying!

Bobby: So can I! It’s that magic crystal that did it!

Variants

(1)
  1. “Pre-Superman Comic Books,” |pending byline|, in New Fun Comics 1, February 1935.