anglesharp / anglesharp Goto Github PK

Hi,I've use AngleSharp version 0.4 for months, and it is robust and works very well for so long, until today when I tried to change the Style of a table row.

My project crashed at this line:
tr.Style.Display = "none";
where tr is of type HTMLTableRowElement. The error message is as below:

A first chance exception of type 'System.NullReferenceException' occurred in AngleSharp.DLL
Additional information: Object reference not set to an instance of an object.

Probably it is something wrong in the internal, because I can inspect that the Display property is not null and has the value of "inherited".

btw, great job!

This is a fantastic project and I can definitely think of more than one place I'm interested in applying it in the not-too-distant future! What are your thoughts on the amount of work to take what you've done and render it to a surface? One application I can see for it is as the basis for user interface rendering for .Net-based games. Currently, to leverage the power of HTML and CSS for a game, you really need to embed something like Awesomium or CEF. Having a completely managed library available that can do all the work of rendering to an offscreen surface and being able to process a stream of input events (either simulated or otherwise) would be really powerful.

Consider this snippet.

var doc = DocumentBuilder.Html("");
var children = new[] {
    doc.CreateElement("span"),
    doc.CreateElement("em")
};
Console.WriteLine(children.Length);
doc.Body.Append(children);
Console.WriteLine(doc.Body.ChildNodes.Length);
Console.WriteLine(doc.Body.InnerHtml);

One might expect that it would print

2
2
<span></span><em></em>

But it actually prints

2
4
<span></span><em></em><span></span><em></em>

hi,
in Html Agility Pack there is a command " HtmlToText()" to strip out Html Tags and return only the raw text, how can we do this here.

thanks in advanced.

First of all, let me congratulate you for AngleSharp. It's a true .net gem with clear goals, roadmap and implementation.

I was playing around with AngleSharp 0.7 and Jint, trying to get an existing website to run a specific javascript which creates an iframe on the document.
I finally did not make it work, probably because 0.7 or the "master" branch are not js ready yet. Despite that, I am summarizing below the items that troubled me and that I see are not changed for the upcoming release:

1. The Navigator property of the AnalysisWindow can't be set as there is no setter. This forced me to specifically implement IWindow.Navigator ontop of AnalysisWindow just to put some value there.
2. The ContentWindow of the IHtmlInlineFrameElement can't be set too. This could not be overridden so I did some monkey workaround in DomFunctionInstance.Call to return the content window of the parent document.
3. Events are implemented in "master" but I see there is no "new Event()" implementation. According to MDN, every browser but IE has it. I am not sure this is in the standard or not.
4. Trying "window.foo = 'bla'; console.log(window.foo);" results in javascript error because of the way window is set as the context in JavaScriptEngine.Evaluate. I worked around this by not returning a new DomNodeInstance from "ToJSValue" in DomFunctionInstance.Call if the "_method.Invoke" result is reference equals to "node.Value". Instead I returned node.Value directly.
5. DomNodeInstance does not have a prototype nor set as extensible nor declares its own Class name. This results in some weird Jint exceptions.
6. The conversions in AngleSharp.Scripting.Extensions.cs are too type safe. A simple "window.setAttribute('allowfullscreen', true)" instead of 'true' results in type cast exception.
7. Finally, I did not find a way to raise the readystatechange event, so I gave up!

I know most of these are not there because I was exploring a half implemented area, however I felt it would be useful to write them down and let you know. I don't consider these bugs so I wouldn't be surprised if you closed this ticket all together as being premature.
Again, thank you for this wonderful project. I don't know the eta nor the milestones of the upcoming release but I am looking forward to it!

namespace AngleSharp.Extensions
{
    using System;
    using System.Linq;
    using AngleSharp.DOM.Css;
    using AngleSharp.DOM;
    using System.Collections.Generic;
    public static class QueryExtensionsEx
    {
        public static List QueryRulesAll(this IDocument document, ISelector selector = null)
        {
            var result = new List();

            document.RulesIterator((CSSRuleList rules) =>
            {
                foreach (var rule in rules.OfType())
                {
                    var cssStyleRule = rule as ICssStyleRule;
                    if (selector == null || (cssStyleRule != null && cssStyleRule.Selector.Text == selector.Text))
                        result.Add(rule);
                }
                return true;
            });

            return result;
        }
    }
}

internal static bool RulesIterator(this IDocument document, Func func)
        {
            var stylesheets = document.Head.Children.OfType().Where(hle => hle.Sheet != null).Select(hle => hle.Sheet).Concat(document.StyleSheets);
            foreach (var stylesheet in stylesheets)
            {
                var sheet = stylesheet as CSSStyleSheet;

                if (sheet != null && !stylesheet.IsDisabled)
                {
                    var rules = (CSSRuleList)sheet.Rules;
                    var continueIterate = func(rules);
                    if (!continueIterate)
                        return false;
                }
            }
            return true;
        }

Hi
The css parser doesn't pick pseudo element '::selection' because of the following. I set the state back to State.Data. Was this your design interntion ?

Boolean OnPseudoElement(CssToken token)
        {
            if (token.Type == CssTokenType.Ident)
            {
                state = State.Data;  ///  Added. This seems to move the parser along ??
                var data = ((CssKeywordToken)token).Data;

                switch (data)
                {
                }
         }

It seems that if you clone an element, remove it from the document, and then re-add it (append it), that the element itself will have the correct Parent, but none of it's children (or grandchildren) have any parent at all. This seems wrong - unless I am misunderstanding something.

I was attempting to use this to use an element (and it's contents) as a template, that I could use to stamp out modified copies - but I also need to get at the parents of the children of the cloned elements.

Here is a failing test:
Gist: https://gist.github.com/jglinsek/dab7f950a3054cf97301

public void replace_the_parent_with_a_clone_and_the_children_of_the_cloned_parent_should_have_a_parent()
{
    const string html = @"
<html>
<body>
    <div class='parent'>
        <div class='child'>
        </div>
    </div>
</body>
</html>
";
    var doc = DocumentBuilder.Html(html);
    var originalParent = doc.QuerySelector(".parent");

    //clone the parent
    var clonedParent = originalParent.Clone();
    clonedParent.Parent.ShouldBe(null);

    //remove the original parent
    var grandparent = originalParent.Parent;
    originalParent.Remove();
    originalParent.Parent.ShouldBe(null);
    grandparent.ShouldNotBe(null);

    //replace the original parent with the cloned parent
    grandparent.AppendChild(clonedParent);
    //the clone itself has the correct parent
    clonedParent.Parent.ShouldBe(grandparent);

    //all the children (and grandchildren) of the cloned element have no parent?
    var cloneElement = (IElement) clonedParent;
    cloneElement.FirstChild.Parent.ShouldNotBe(null); //FAILS
}

Hi,

this project looks awesome, would you kindly add some documentation on usage.

thanks in advanced.

If input document is encoded in utf8 and contains html elements with 'lang' attribute, value of those elements is mojibake'd
To reproduce this bug I just copied text from MS Word to clipbard and tried to parse resulted html thru AngleSharp library.
Word generates an html with copied text as follows:

<html>
<head>
<meta http-equiv=Content-Type content="text/html; charset=utf-8>
</head>
<body lang=RU style='tab-interval:35.4pt'>
<!--StartFragment-->
<p class=MsoNormal><span lang=EN-US style='mso-ansi-language:EN-US'>тест</span><o:p></o:p></p>
<!--EndFragment-->
</body>
</html>

To simplify, I removed all extra meta-information about styles and whatever from this sample html.
Text inside span has attribute lang and entire document is already encoded with utf. While traversing node-tree I receive wrong text inside span.

AngleSharp returns wrong value of alfa channel If content has specified background property in rgba format.
Example:
<table style="border:none;border-collapse:collapse;">
<tr>
<td align="right" valign="top" style="border: 1px solid #000;background:rgba(128,128,128,255);">
Test
</td>
</tr>
</table>

if (fragment = (!onlyQuery && input[index] == Symbols.Num))

Draft interfaces for optional resource and rendering defined like the ones already in their for the IoC container, or the HTTP requester.

Deserialization + serialization of a stylesheet should result in exactly the original stylesheet, including all whitespaces. This is an important guarantee in real-world editing - the user's formatting should not be messed up.

1. If portions of a stylesheet are modified, only the formatting of those portions should be changed.
2. All content from the source should be represented by something in the DOM. As an example of what I mean see this description of how Roslyn represents .net source code.

A straightforward way of achieving this is to store the original parsed text of any ICssObject. Then when ToCss() is called it can just return the text instead of recomputing it. Recomputing is only required if the ICssObject is modified. (As an optimization, rather than copying the text of the original document into every element, one can keep the entire document as a string and reference sections of the string from within the ICssObject. But this is just a possible implementation detail.)

Hi,

I found a bug. When using QuerySelectorAll and using nth-child with calculations, for example tr:nth-child(n+2) it does not work correctly. It returns all children instead of the ones from second node on.

Hey guys, just came across this. Any benefit for SimpleBrowser? https://github.com/FlorianRappl/AngleSharp

Parse and build CSS media queries expressions.

HTMLSelectElement should expose Options, selectedIndex script properties.

Small grammatical ? Thanks

I need to convert a document to such a format that it should be both HTML and XML in the same time. That is, the markup like <select/> is not allowed, should be <select></select> instead, but the markup like <img src="..." /> is required. Would be nice if AngularSharp could provide some control over the process of DOM serialization.

Html Agility Pack has something like this. At least, it can output XHTML.

Just downloaded version 0.6.0 and found the following issue after a quick try:

For a simple style definition, the CssText property shows empty

string css = @"html{font-family:sans-serif;}";
var stylesheet = AngleSharp.Parser.Css.CssParser.ParseStyleSheet(css);

foreach (var rule in stylesheet.Rules)
{
Console.WriteLine(rule.CssText);
}

The end result ends up something like this: html {\r\n;;}

It used to work fine in version 0.5.1.

Anything I did wrong here?

Test code is here:

internal class Program
{
    private const string TestUrl = @"http://www.baidu.com";
    private static readonly Regex TitleRegex = new Regex("<title>(.*?)</title>");

    private static void Main(string[] args)
    {
        var angleSharpTitle = DocumentBuilder.Html(new Uri(TestUrl, UriKind.Absolute)).Title;
        var htmlAgilityPackTitle = new HtmlWeb().Load(TestUrl).DocumentNode.SelectSingleNode("//title").InnerText;
        var webClientTitle = WebClientDownload();
    }

    private static string WebClientDownload()
    {
        return
            TitleRegex.Match((new WebClient() {Encoding = Encoding.UTF8}.DownloadString(TestUrl)))
                .Groups[1].Value;
    }
}

www.baidu.com is a website in China. In the test code, HtmlAgilityPack and WebClient are both correct, but AngleSharp does not, one character is not display correctly.

I've used AngleSharp(version 0.50 or older) for a small Windows Store app project for a quite long time(more than a half year) and it works great and stable. But when I today update it to version 0.70, I encountered two minor problems.

1. DocumentBuilder.Html(Uri uri) method won't work for Windows Store app project. Following is the code:

   static void Test()
   {
       IDocument doc = DocumentBuilder.Html(new Uri("http://stackoverflow.com/", UriKind.Absolute));
       Debug.WriteLine(doc.ToHtml());
   }
   

The compiler complains:

"A first chance exception of type 'System.InvalidOperationException' occurred in mscorlib.dll

Additional information: The API 'System.Net.HttpWebRequest.set_UserAgent(System.String)' cannot be used on the current platform. See http://go.microsoft.com/fwlink/?LinkId=248273 for more information."

1. DocumentBuilder.Html(string pageSource) can't correctly parse page source which contains special charset. For example:

   static async void Test2()
   {
       HttpClient httpClient = new HttpClient();
       IBuffer response = await httpClient.GetBufferAsync(new Uri("http://item.jd.com/11312278.html"));
   
       string pageSource = Encoding.GetEncoding("gb2312").GetString(response.ToArray(), 0, (int)response.Length - 1);
   
       Debug.WriteLine("original pagesource == "+pageSource); //correct output
   
       IDocument doc = DocumentBuilder.Html(pageSource);
       Debug.WriteLine("parsed html text == " + doc.ToHtml()); //incorrect output(Chinese characters all display as unreadable special characters)
   }
   

These two problems are all new, maybe it's due to some minor changes in the new version.

According to this section of the css specification the correct term for property/value pairs (including the important flag) is Declaration, not Property. AngleSharp currently names this class CssProperty. This is confusing, especially since CSSStyleDeclaration has this:

List<CSSProperty> _declarations;

Suggest changing the name of CSSProperty to CssDeclaration. Note the use of CamelCase - I prefer Css to CSS, in keeping with .net coding conventions. AngleSharp seems divided on the issue :)

Edit: Looks like this (incorrect) naming is used for all derived declaration types as well, so this might be a bigger change. However, that only makes it all the more needed. Hopefully Visual Studio can make the renaming easy.

Currently, it is not possible to instantiate a HTML Element for example HTMLInputElement as its constructor declared with internal modifier. Sometimes we may need to create and add dynamic elements to DOM. Please change all the HTML elements constructors so that they can be added dynamically. HtmlDocumet.AppendChild is of no use in this case.

Implement methods to validate parsed XML documents.

I am trying to compare the computed styles from Chrome and IE with AngleSharp. Running into the above issue.

I'm founding a html parser for wp8.1. AngleSharp is powerful, but I could not use it with wp8.1. Please support wp8.1. Thanks.

The error is thrown when trying to retrieve a web page via HtmlAsync in a Windows Phone project. I am using version 0.5 via NuGet.

Example:
var document = await AngleSharp.DocumentBuilder.HtmlAsync(new Uri("http://stackoverflow.com"));

Form property is missing in HTMLTextAreaElement.

Hi, I am new to AngleSharp. First of all, I would like to say to the author, Great Job, thanks!

I am using AngleSharp to write a small Windows Store app. I've tried to call DocumentBuilder.HtmlAsync() to get a html document object, but I got a runtime error:

A first chance exception of type 'System.InvalidOperationException' occurred in mscorlib.dll
An exception of type 'System.InvalidOperationException' occurred in mscorlib.dll but was not handled in user code
Additional information: The API 'System.Net.HttpWebRequest.set_UserAgent(System.String)' cannot be used on the current platform. See http://go.microsoft.com/fwlink/?LinkId=248273 for more information.

Here is what I am doing. Am I missing something?

        AngleSharp.Configuration.UseDefaultHttpRequester = true;
        var document = await AngleSharp.DocumentBuilder.HtmlAsync(new Uri("http://stackoverflow.com/questions/20115672/how-does-the-static-type-affect-this-code"));

My developing environment is Windows 8.1 and Visual Studio 2013 express for Windows Store app.
Thanks

The constraint where TNode : INode makes it impossible to write something like this:

document.Body.Descendents<IUrlUtilities>()

which could be convenient if IHtmlImageElement implemented IUrlUtilities (it doesn't though).

Hi
There is a 'HttpWebResponse' memory leak, as this not disposed. I think you are holding onto to 'Content" stream. The current implementation does not scale beyond few external link requests to, say, style sheets.

I modified thus:

             DefaultResponse GetResponse()
            {
                if (_response == null)
                    return null;

                var result = new DefaultResponse();
                var headers = _response.Headers.AllKeys.Select(m => new { Key = m, Value = _response.Headers[m] });
                result.Content = new System.IO.MemoryStream(); 
                _response.GetResponseStream().CopyTo(result.Content);
                result.Content.Position = 0;
                result.StatusCode = _response.StatusCode;
                result.Address = new Url(_response.ResponseUri);

                foreach (var header in headers)
                    result.Headers.Add(header.Key, header.Value);

                _response.Dispose();
                _response = null;
                return result;
            }

Can the css parser parse unknown properties and return them as simple string values to be handled by calling code? From my (very limited) look into AngleSharp, it seems that properties with unknown names are being discarded.

Alternatively, is it possible to pass in a handler for properties that AngleSharp does not know? The idea is the handler parses and instantiates the correct CssProperty subtype for properties it cares about.

HTML table element missing Rows property.
HTML row element missing Cells property.

Hello,

do you plan to integrate javascript support (V8 Js engine) ?
Or this library will expose dom only and no script support supposed?

Thanks

It's something that is natural to expect. And the newer .NET APIs have it. For example

XElement.Parse("<x/>").ToString()

gives us

<x />

I would expect an alike behaviour from AngleSharp.Url, IHtmlElement etc.

Url url = new Url("https://www.google.de/");
url.IsInvalid //true

It looks like that there's a missing negation in Url.cs line 404:

_error = ParseScheme(input.Trim());

CSS model implemented (e.g. getComputedStyle works)

The table element does not automatically close an open paragraph. See #46 for the initial hint.

There is some logic errors with line 119 & 122 in file AngleSharp/DOM/Css/Properties/Border/CSSBorderProperty.cs, probably caused by copy and paste. Also border style definition like "border: 1px outset currentColor" is not well supported.

The task is to change a bit some of the relative URLs in the document.

var document = DocumentBuilder.Html("<a href='project/part2'>text</a>");
var a = (IHtmlAnchorElement)document.Body.FirstElementChild;
a.Href += "/1";
Console.WriteLine(document.Body.InnerHtml);

Expected output: <a href="project/part2/1">text</a>
Actual output: <a href="about:///project/part2/1">text</a>

Is it supposed to be like this? What's the best way to work this around?

Configuration of AngleSharp will be easier, more adaptive and flexible.

In your design principle of the above, we need like the following, so no dependency on IWindow

CSSStyleRule/CSSRule
internal override void ComputeStyle(CssPropertyBag style, IWindow window, IElement element)
        {
            if (_selector.Match(element))
                style.ExtendWith(_style, _selector.Specifity);
        }
        internal override void ComputeStyle(CssPropertyBag style, IDocument document, IElement element) 
        {
            if (_selector.Match(element))
                style.ExtendWith(_style, _selector.Specifity);
        }

StyleExtensions

/// 

        /// Inherits the unspecified properties from the element's parents.
        /// 
        /// The bag to modify.
        /// The element that has unresolved properties.
        /// The associated window object.
        public static void InheritFrom(this CssPropertyBag bag, IElement element, IDocument document)
        {
            var parent = element.ParentElement;

            if (parent != null)
            {
                var styling = document.computeDeclarations(parent); 

                foreach (var property in styling.Declarations)
                {
                    var styleProperty = bag[property.Name];

                    if (styleProperty == null || styleProperty.IsInherited)
                        bag.TryUpdate(property);
                }
            }
        }

internal static CSSStyleDeclaration computeDeclarations(this IDocument document, IElement element)
        {
            var bag = new CssPropertyBag();

```
        document.RulesIterator((CSSRuleList rules) =>
        {
            rules.ComputeStyle(bag, document, element);
            return true;
        });

        var htmlElement = element as HTMLElement;

        if (htmlElement != null)
            bag.ExtendWith(htmlElement.Style, Priority.Inline);

        bag.InheritFrom(element, document);
        return new CSSStyleDeclaration(bag);
    }
    public static ICssStyleDeclaration ComputeDeclarations(this IDocument document, IElement element)
    {
        return document.computeDeclarations(element);
    }
```

Start writing classes for various elements in the MathML and SVG DOM.

Hello.
Sorry about my English.

I have job , download and parse data from about 30000 web pages.

In parse Method , I use AngleSharp like this :

public override IEnumerable<DIRTY_SCHEDULE> Fetch(string ctx, string url = "") {
    var dom = DocumentBuilder.Html(ctx);
    //不支持 even
    //var trs = dom.QuerySelectorAll("#accordion2 table tbody tr:even");
    var trs = dom.QuerySelectorAll("#accordion2>.accordion-group>.accordion-heading>table>tbody>tr");
    for (var i = 0; i < trs.Length; i = i + 2) {
        var tr = trs[i];
        var tds = tr.QuerySelectorAll("td");
        var entry = new DIRTY_SCHEDULE {
            CARRIER = tds[0].Text(),
            ROUTE = tds[1].Text().Trim(),
            VESSEL = tds[2].Text().Trim(),
            VOYAGE = tds[3].Text().Trim(),
            ORGIN = tds[4].Text().Trim(),
            ETD = tds[5].Text().Trim().ToDateTime("yyyy-MM-dd", DateTime.Now),
            DEST = tds[6].Text().Trim(),
            ETA = tds[7].Text().Trim().ToDateTime("yyyy-MM-dd", DateTime.Now),
            TT = tds[8].Text().Trim().ToDecimalOrNull(),
            DIRTY_SCHEDULE_TRANSF = this.FetchTransf(trs[i + 1]).ToList(),
            SOURCE = url,
            APP = "Fetcher.Soushipping",
        };

        entry.UNQTAG = entry.GetUNQTag();

        yield return entry;
    }
}

private IEnumerable<DIRTY_SCHEDULE_TRANSF> FetchTransf(IElement tr) {
    var tbls = tr.QuerySelectorAll("table.widget");
    //第一个列出的是起始地
    for (var i = 1; i < tbls.Length; i++) {
        var rows = tbls[i].QuerySelectorAll("tr");
        if (rows.Length == 3)
            yield return new DIRTY_SCHEDULE_TRANSF {
                VESSEL = rows[0].Text().Trim(),
                AT = rows[1].QuerySelector("td").Text().Trim(), //rows[1].FirstChild.Text().Trim(),
                VOYAGE = rows[2].Text().Trim(),
                SEQ = i - 1
            };
    }
}

It work fine.
But, Memory leaked very serious.

You can see it from here :

3:20 1.8G!
What's wrong ?
IElement not disposed ?

But use HtmlAgilityPack :

3:20 only 270 M.

The code use HtmlAgilityPack:

public override IEnumerable<DIRTY_SCHEDULE> Fetch(string ctx, string url = "") {
    var doc = new HtmlDocument();
    doc.LoadHtml2(ctx);
    var root = doc.DocumentNode;
    var trs = root.QuerySelectorAll("#accordion2>.accordion-group>.accordion-heading>table>tbody>tr")
        .ToList();
    for (var i = 0; i < trs.Count(); i = i + 2) {
        var tr = trs[i];
        var tds = tr.QuerySelectorAll("td").ToList();
        var entry = new DIRTY_SCHEDULE {
            CARRIER = tds[0].InnerText.Clear(),
            ROUTE = tds[1].InnerText.Clear(),
            VESSEL = tds[2].InnerText.Clear(),
            VOYAGE = tds[3].InnerText.Clear(),
            ORGIN = tds[4].InnerText.Clear(),
            ETD = tds[5].InnerText.Clear().ToDateTime("yyyy-MM-dd", DateTime.Now),
            DEST = tds[6].InnerText.Clear(),
            ETA = tds[7].InnerText.Clear().ToDateTime("yyyy-MM-dd", DateTime.Now),
            TT = tds[8].InnerText.Clear().ToDecimalOrNull(),
            DIRTY_SCHEDULE_TRANSF = this.FetchTransf(trs[i + 1]).ToList(),
            SOURCE = url,
            APP = "Fetcher.Soushipping",
        };

        entry.UNQTAG = entry.GetUNQTag();

        yield return entry;
    }
}


private IEnumerable<DIRTY_SCHEDULE_TRANSF> FetchTransf(HtmlNode tr) {
    var tbls = tr.QuerySelectorAll("table.widget").ToList();
    //第一个列出的是起始地
    for (var i = 1; i < tbls.Count(); i++) {
        var rows = tbls[i].QuerySelectorAll("tr").ToList();
        if (rows.Count == 3)
            yield return new DIRTY_SCHEDULE_TRANSF {
                VESSEL = rows[0].InnerText.Clear(),
                AT = rows[1].QuerySelector("td").InnerText.Clear(), //rows[1].FirstChild.Text().Trim(),
                VOYAGE = rows[2].InnerText.Clear(),
                SEQ = i - 1
            };
    }
}

var document = DocumentBuilder.Html("");
var nodeArray = DocumentBuilder.HtmlFragment("<br>").ToArray();
document.Body.Prepend(nodeArray);
Console.WriteLine(document.ToHtml());

prints

<html><head></head><body><html><head></head><body><br></body></html></body></html>

I expected to get:

<html><head></head><body><br></body></html>

Helpers for URL management, easier DOM modifications, etc. would be nice.

Using the AngleSharp html parser on the file below causes an invalid attribute node to be created.
The attribute name is something line "\documentary".

(Btw, the content is from a news article in the Irish Times.)
https://www.dropbox.com/s/9joo87xf05ssynw/error_in_attribute.html?dl=0