take it to autozone and have them do a check on your battery, starter, and fuel pump. i had them check my battery, it failed, alternator came out fine. i was happy about that. but they will tell you if anything is wrong.
if it was only happening warm i would say vapor lock. but since its cold or warm im not sure.
mine on the first start of the day took a good bit, but for the rest of the day was almost instant.