Results

It is difficult to evaluate this model, because there is no clear metric to measure the “realness” of images.

Here some of the results I get on my validation set:

alt text

I have tried to generate with false captions, to see if the generator have learnt to use them :

alt text

We not see some clear differences. I think the conditioning on the caption is not significant in this generator model. However, the loss Ld_fake_captions go to zero on the discriminator, that mean that the discriminator predict the reject class (0) for images from the dataset with an invalid caption. The discriminator is then able to use the information contained in the embeddings. Then the generator inability to use them is a problem in the generator architecture. In future work, I will address this issue.

But we can instead experiment a variation on the conditioning to observe the dependance the model have learn on it.

Observation

Room pictures

The rooms pictures are well reconstructed, maybe because the rooms pictures are geometric with fairly simple structure. The other possibility is that the training set have a lot of room related pictures. The vehicles are also well reconstructed.

A bedroom area with bed, window and two nightstands..jpg A room with a couch, table, lamp and a chaise..jpg A small black car is sitting on the pavement.jpg

Visage

The model have difficulties to generate visage and complex pattern. Maybe I need to train more the model or augment the layers numbers to allow it to express more complex structures.

A boy in white shirt and glasses eating food..jpg A group of zebras and a giraffe in a pen..jpg Two zebras standing on a sidewalk near passing cars with arms sticking out..jpg

More results

Here more results with the associated captions in tooltip.

An airplane with two propellers near a grass and a bench..jpg A female skateboarder carrying a purse followed by two bicyclists..jpg a man walking out of the water with his surfboard.jpg An orange backhoe atop the bed of a white truck..jpg A very long train is stopped inside of a building ..jpg A white plate topped with a pizza cut into 6 slices..jpg A trainer in the woods with a large elephant.jpg A black and white desk with a laptop and two speakers..jpg A man is eating a doughnut and toys are on the table..jpg a double bus driving through a parking lot.jpg a little boy that is standing up with a glove.jpg Man in a tiger suit in front of another man on the phone.jpg A baby in a high chair with a party hat has a cake on the tray..jpg People on snowboards or skis on a snowy hill..jpg A skateboarder riding past some cones out side..jpg A toilet and a trash can in a room..jpg A small jet plane flying in some dark skies..jpg The surfer is hanging on to his surf board and his kite..jpg Several different vases of varying shapes and sizes are seen here..jpg A dish with lettuce and barbecue chicken being served to someone..jpg a blue bus sitting by the side of the road.jpg A woman eats out of a bowl in a large dining room..jpg A bedroom area with bed, window and two nightstands..jpg a plate of food with a sandwich and french fries on a table.jpg there is a very tall building that has a very pointed top.jpg A group of young children sitting next to each other on a grass field..jpg A zebra grazing on food behind a metal fence..jpg Two motorcyclists are jumping their bikes high into the air..jpg The young woman visiting the zoo is posing for a photo with a statue of a giraffe..jpg A group of zebras and a giraffe in a pen..jpg A room with a couch, table, lamp and a chaise..jpg A pasta dish with sides of green beans and oranges..jpg A small airplane sitting on the tarmac at an airport..jpg A person cutting a pizza with a circular pizza blade..jpg a green shuttle bus parked at a sign.jpg A person is hanging from a rope with a board attached to his feet..jpg Three elephants standing together in a wild, grassy area..jpg A player is fallen on the ground and appears injured..jpg Two young girls lay in bed using laptops..jpg A large plane parked at the gate for unloading..jpg Light streams into an apartment through the patterned curtain..jpg A person para sailing in the ocean with a person fishing from the shore..jpg A bunch of red and orange umbrellas in a mall..jpg A sandwich consisting of crusty bread, greens, ham, and pickles, on a plate with french fries..jpg A couple of kids are sitting on a surfboard.jpg A MAN IN THE ARMY STANDING HOLDING THREE DOGS.jpg two small boats in a large body of water.jpg A baseball team sitting on benches in black and white..jpg A group of women playing a video game..jpg A woman skiing down the side of a snow covered mountain..jpg Elderly adults standing in kitchen with modern appliances..jpg Small dog peeking out of a basket on a motorcycle..jpg Tables at a fair one showing cakes up for competition..jpg A toddler is stuffing a banana in his mouth..jpg A brown cow standing on top of a smaller brown cow next to a building..jpg A small black car is sitting on the pavement.jpg A small dog sits on a bed with striped sheets..jpg A group of motorcycle riders traveling down a shaded street..jpg The young woman is laying on the unmade bed..jpg Two skiers on at the top of a mountain slope..jpg A man sitting on a bench next to a brown cow with horns..jpg Two blue street signs pointing in opposite directions.jpg A image of a boy swinging a baseball bat..jpg A small apartment living room on a sunny day..jpg Three adult and two child skiers posing on a slope..jpg A sign is prominent on a street corner..jpg A man riding a dirt bike on top of a pile of dirt..jpg A green older bus riding down the street..jpg A cat curled up sleeping on a bed..jpg A vase with several different species of flower sits on a glass table..jpg A school bus makes its way around a turn..jpg A landscape shows a plowed field and the top of a motorcycle..jpg Lady standing in the rain with an umbrella over her head..jpg A bright kitchen with tan and blue cabinets..jpg a fire hydrant spewing water on a city side walk.jpg A woman with a necklace taking a hotdog and smiling.jpg Fancy motorcycle parked for display at motor vehicle street fair..jpg A wooden park bench sitting on top of a park..jpg a man is helping a child on a skateboard.jpg Two bottles of red wine on a table with a glass of wine..jpg A man In a living room standing playing a wii.jpg A person walks with a surfboard toward a body of water..jpg some kind of food that is on a plate.jpg A young man is throwing a Frisbee on the beach..jpg a kitchen with a table by a window and a desk by a fridge.jpg Small narrow city street with paved walkway..jpg A red, yellow and gray train at a station.jpg Two people are walking on the street with an umbrella..jpg This sunny bedroom features dark wood furniture and a bedspread with bold geometric design.jpg A red and white fire truck driving in a parade..jpg This boy is riding his skateboard on a sidewalk..jpg a person sharpening a knife by hand in a kitchen.jpg Spectators watching two people collide with a frisbee..jpg A dried black flower in a long, tall black & white vase..jpg Village women are looking at bunches of bananas..jpg A toilet is in the corner of a clean bathroom..jpg A row of sail boats with trees in background..jpg Two sinks in a kitchen with dirty dishes in them..jpg Lady with a slice of piece in front of a stack of pizza boxes..jpg A boy in white shirt and glasses eating food..jpg A plate that is full of various types of food..jpg Skiers holding ski poles traveling a snowy path..jpg A young man balanced on his skateboard..jpg A cat under the mesh lining of an open suitcase..jpg a bus sits parked as a man stands in front of it.jpg a plate of fish , fries and coleslaw..jpg A bathroom scene with a row of white toilet..jpg A pink shirted girl riding on an elephant..jpg A forlorn, black and white photograph of an abandoned train..jpg a toddler playing with a piece of paper in her hands.jpg A shadow on a brick wall lined with old toilets..jpg A pack of hotdogs are on the grill while a plate of them already cooked are on a plate on the side along with buns and about 5 other packs of weiners..jpg A couple of people on a motor bike on a street..jpg a person lays down while using a cell phone.jpg A man that is standing on a skateboard on a bench..jpg A train pulls into a train station at 12:30 pm..jpg A woman sitting on a brown sofa by pictures.jpg A sporting event outside as well as some cups of juice and food on a plate..jpg A crowd of people stand by canopy tents in the rain..jpg A cat dressed in a bamboo hat looks just beyond camera..jpg A room with large windows, a couch and a table..jpg There are people standing together under a umbrella.jpg A black and silver motorcycle parked on the pavement.jpg A skateboarder is riding his skateboard on a railing..jpg A male tennis player reaches to return the ball..jpg A man is putting a mug into the microwave..jpg American Airlines plane flying in a clear sky,.jpg A sandwich on a plate with a glass of wine sitting on a dinner table..jpg A man pulling a tie out of his mouth..jpg Motorcycle parked next to SUV in car garage.jpg a small group of zebra and giraffe in an enclosure.jpg A cop car parked next to a fire hydrant with a hose sticking out of it..jpg A group of young kids getting ready to ski on a mountain..jpg A giraffe standing in a cage at a zoo..jpg A woman and children are watching television from a couch..jpg A woman dressed in red cowgirl gear rides a horse through a parade..jpg A red fire hydrant surrounded by lush foliage..jpg A black railroad train approaching the train platform.jpg Old disassembled train cars sit idle on train tracks.jpg Two buns holds meat and coleslaw as the sit on white papers..jpg A baseball player swinging a base ball at at home plate..jpg Two zebras standing on a sidewalk near passing cars with arms sticking out..jpg A do not enter sign next to a one way and street sign..jpg A giraffe stands alone the shade in an enclosure..jpg A room that has a table with chairs around it.jpg Two stuffed animals with two cupcakes on a blanket..jpg A kitchen with an island and bar stools..jpg The top of a steeped church building with clocks and small windows..jpg A cell phone is on and connected to cords..jpg A stop sign with other street signs on top of it..jpg A banana and two oranges are placed in a bowl..jpg A very cute brown dog in a basket on a bike..jpg A laptop sitting on a table near papers and cups..jpg there are many people standing outside of a plane.jpg The young girl and her dog are laying on the bed..jpg A kitchen is shown with an oven and stove..jpg A Pakastan airline jet sitting on the tarmac.jpg The group of people in business suits is standing beside a large poster..jpg A crowd observes a man swing his racket during a tennis match..jpg A young man slicing up a pizza on a counter in his kitchen.jpg ’