North Korea pins the blame on John Bolton and says it will not be forced to give up its nuclear weapons.
Mr Pompeo said there had been "good and productive conversations" with Mr Kim.
At a summit in April, Kim Jong-un and South Korean President Moon Jae-in vowed to work to rid the peninsula of nuclear weapons, although the means of achieving this were not detailed and previous such pledges have been abandoned.
US President Donald Trump has welcomed home three American detainees released by North Korea.
A new world. But did someone get there first?
The three men, Kim Hak-song, Tony Kim and Kim Dong-chul, were smiling and waving and appeared in good health.
Asked if this was his proudest achievement, he said that would be "when we denuclearise that entire peninsula".
Mr Trump said he hoped he could travel to North Korea one day and that he believed Mr Kim wanted to bring his country "into the real world".
Who are North Korea's American detainees?
"It's a great honour. But the true honour is going to be if we have a victory in getting rid of nuclear weapons."
People can be jailed by the regime for almost anything, activists say, with crimes ranging from watching a South Korean DVD to trying to defect from the country.
The main issue, of course, will be North Korea's nuclear weapons, which Mr Trump has demanded Pyongyang give up.
The date and location have been decided but not yet announced.
The White House said the trio had been freed as a gesture of goodwill ahead of the planned meeting between Mr Trump and North Korean leader Kim Jong-un.
Share this with
The trio had been jailed for anti-state activities and placed in labour camps.
Mr Trump and his wife Melania entered the plane after it landed at about 02:45 local time (06:45 GMT) and a few minutes later emerged with the three men to wave to the media.
In an impromptu chat before the media with Mr Trump, Kim Dong-chul said: "It's like a dream and we are very, very happy. We were treated in many different ways. For me, I had to do a lot of labour. But when I got sick I was also treated by them."
The president said he appreciated that Mr Kim had allowed the men to leave as "frankly we didn't think that was going to happen before the meeting".
Sign up for our newsletter
Political prisoners are often sent to separate prisons - usually brutal labour camps, which involve difficult physical work such as mining and logging.
The convictions of the trio were widely condemned as political and an abuse of human rights.
"We thank God and all our families and friends who prayed for us and for our return."
His parents, Fred and Cindy Warmbier, said they were "happy for the hostages and their families. We miss Otto".
The last American to be freed - Otto Warmbier, who was jailed for stealing a hotel sign - was released last year but was fatally ill, and died shortly after returning home.
Mr Trump said it was a "special night for these really great people" as they arrived at the Andrews Air Force Base near Washington.
Of the upcoming summit, he said: "I think that we're going to have... a very big success... I really think we have a very good chance of doing something very meaningful."
The three had released an earlier statement saying: "We would like to express our deep appreciation to the United States government, President Trump, Secretary [of State Mike] Pompeo and the people of the United States for bringing us home.
There has been no word from North Korea on its agenda or what it might offer, although a key issue is certain to be the presence of 30,000 US military personnel in South Korea, along with the removal of sanctions.
It came during a visit to Pyongyang by Mr Pompeo to arrange details of the meeting between Mr Trump and Kim Jong-un.
Copy this link
He said the talks would not be held at the Demilitarized Zone (DMZ) that separates North and South Korea, one of the suggested venues, and Singapore remains the most likely option.
Speaking at Andrews Air Force Base, Mr Trump said he wanted to pay his "warmest respects" to the couple.
On behalf of the American people, WELCOME HOME! pic.twitter.com/hISaCI95CB
American missionary Kenneth Bae, who was sentenced to hard labour, later said he was forced to work on a farm six days a week despite being in poor health.
These are external links and will open in a new window
Mr Trump says the time, date and location will be announced "within three days".
"We're planning it will be a single day, but in the event that there is more to discuss, there'll be an opportunity for it to extend into the second day as well," Mr Pompeo said.
Mr Trump said he thought the summit would be "a big success".
End of Twitter post by @realDonaldTrump
Mr Kim said he had accepted a US proposal to grant the three detainees an amnesty, adding that his meeting with President Trump would be an "excellent first step" towards improving the situation on the Korean peninsula, according to the North Korean state news agency KCNA.