Get HTML Code from a website after it completed loading

c# get html from url
how to read web page content in c#
javascript load html file
javascript execute after all scripts loaded
javascript wait for page to load
puppeteer sharp html to pdf
.net core html parser

I am trying to get the HTML Code from a specific website async with the following code:

var response = await httpClient.GetStringAsync("url");

But the problem is that the website usually takes another second to load the other parts of it. Which I need, so the question is if I can load the site first and read the content after a certain amount of time.

Sorry if this question already got answered, but I didn't really know what to search for.

Thanks, Twenty


Edit #1

If you want to try it yourself the URL is http://iloveradio.de/iloveradio/, I need the Title and the Artist which do not immediately load.

You are on the wrong direction. The referenced site has playlist api which returns json. you can get information from :

http://iloveradio.de/typo3conf/ext/ep_channel/Scripts/playlist.php

Edit: Chome Inspector is used to find out Playlist link

JavaScript loading even before HTML loads - JavaScript, The code is working fine if i place it as inline in the webpage itself. How should i tell the browser to load HTML first and JavaScript later? Thank You, Sri. for the page. Only after the page has fully completed loading does the onload event fire. Wait til Document is fully loaded (ReadyState = Completed) Get Document content; I guess one the simple ones is to use a WebControl where you navigate to your url and wait for the controls ready or complete state. After that you could start parsing. Here on SO is a solution: htmlagilitypack and dynamic content issue

You could use Puppeteer-Sharp:

await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultRevision);
using (var browser = await Puppeteer.LaunchAsync(new LaunchOptions { Headless = false }))
using (var page = await browser.NewPageAsync())
{
    await page.SetViewportAsync(new ViewPortOptions() { Width = 1280, Height = 600 });
    await page.GoToAsync("http://iloveradio.de/iloveradio/");
    await page.WaitForSelectorAsync("#artisttitle DIV");
    var artist = await page.EvaluateExpressionAsync<string>("$('#artisttitle DIV')[0].innerText");
    Console.WriteLine(artist);
    Console.ReadLine();
}

Tips for authoring fast-loading HTML pages, Other tools can "compress" JavaScript by reformatting the source or by If you use background images a lot in your CSS, you can reduce the number of HTTP Conditional Get for RSS Hackers · HTTP 304: Not Modified · HTTP ETag Any dynamic features that require the page to complete loading before  I hope this simple preloader will help you improve your site’s user experience. If you get any difficulty using the codes to your website then do comment below. Feel free to contact me for any help. Subscribe to our email list to get updates to this post.

If there are things that load after, it means that they are generated by javascript code after page load (an ajax request for example), so no matter how long you wait, it won't have the content you want (because they are not in the source code when it loads).

Easy way to do it:

Use a WebBrowser and when DocumentCompleated event triggers wait till the element you want appears.

The Right Way:

find the javascript yourself and trigger it yourself (easy to say, hard to do).

Selecting dynamically-loaded content, Some webpages show the desired data when you load them in a web browser. When this happens, the recommended approach is to find the data source and extract the data from it. scrapy fetch --nolog https://example.com > response.​html. If The effort is often worth the result: structured, complete data with minimum  Now that you have the index file open in both your browser and your editor, we’ll start writing some code! HTML Tags. Let’s look at some of the basic features of HTML. HTML is made up of tags. Tags are special text that you use to mark up, or distinguish, parts of your web page. Hence the hypertext “markup” language.

HTML DOM Image complete Property, Well organized and easy to understand Web building tutorials with lots of examples of how If the image is finished loading, the complete property returns true. JavaScript loading even before HTML loads you can use the onload method of the window object to run js code after the window has loaded. Only after the page has fully completed loading

I have checked out the website, data is loaded by javascript. You only can get the html using httpClient.GetStringAsync("url");. As far as I know, there is no luck to get the elements what is manipulate by browser.

How a webpage is loaded and displayed, Looking a bit deeper into the nuts and bolts of how a webpage is displayed in a browser. Each of these steps are often performed several times during a page load. let's look at what makes a webpage get requested in the first place. Once the browser has received the document (html file) it reads it. Salvete! I needed, simply, a function I could call to make the code wait for the page to load before it continued. After scouring the web for answers, and fiddling around for several hours, I came up with this to solve for myself, the exact dilemma you present.

Running Your Code at the Right Time, Run your code at the right time by making sense of DOMContentLoaded, load, async, Enter after typing in a URL and, if the stars are aligned properly, your page loads. and takes up a very tiny sliver of time to complete from beginning to end: almost always find yourself in when interacting with your HTML document. Many a times web developers come across situations where there are large time-consuming things that need to be done in the Page_Load event, and we don't obviously want the end-user to see just a plain white screen. A "Loading.." message or a Loading-GIF would be more user friendly. This article tells how to show such a message. Using the Code

Javascript load html file into dom, The required URL parameter specifies the URL you wish to load. time I need to parse HTML is the free and open source HtmlAgilityPack library. Once this process is done, the parsing can resume, and the rest of the HTML can be analyzed  To achieve this, I just wanted to write a quick and dirty front end script, that would check to see if the content was loaded, and show a slightly animated loading screen if it isn’t loaded yet. Here’s my pseudo code: Basically I want to check the DOM, to see if the content has loaded, and if not, I’ll add some “loading” text to the dom.

Optimize Website Speed With Chrome DevTools, Learn how to use Chrome DevTools to find ways to make your websites load faster. You should be able to complete the tutorial still, just keep in mind that your UI In short, when you're working with your own code, the Coverage tab can help display a page, such as parsing and executing HTML, CSS, and JavaScript. If you can view some of your HTML code then we can skip #1-3 above and look within the page being loaded into your browser window. When your view a web page in your browser window the reality is that only one file is being displayed and all content that appears on the page being displayed in your browser window is brought to you through that

Comments
  • Thanks a lot you saved my day, how did you find that one?
  • If this answer is useful, please consider to tick it as answer. thanks
  • Still how did you found out?
  • Check my answer again please