philss / floki Goto Github PK
View Code? Open in Web Editor NEWFloki is a simple HTML parser that enables search for nodes using CSS selectors.
Home Page: https://hex.pm/packages/floki
License: MIT License
Floki is a simple HTML parser that enables search for nodes using CSS selectors.
Home Page: https://hex.pm/packages/floki
License: MIT License
consider this case:
Floki.parse("<ul><li>text1</li><li>text2</li><ul>") |> Floki.text
returns
text1text2
If we give the consumer the possibility to add a separator
Floki.parse("<ul><li>text1</li><li>text2</li><ul>") |> Floki.text (separator: " ")
returns
text1 text2
The current implementation does not allow searching for elements using a multi selector, like a.some-class
or .some-class.another-class
. This is because it is coupled to the idea of one search per selector at time. This means that I can't search a "tag" and a "class" in the same element at the same time.
We need to provide a basic infrastructure to be able to search using groups of selectors.
This could bring a huge flexibility to Floki, since it's would be possible to truly simulates the "jQuery" selector (actually simulates Sizzle, the jQuery query selector engine).
Examples of queries to support:
Floki.find("a.foo")
Floki.find(".foo.bar")
Floki.find(".foo[data-js=bar]")
Floki.find("a.foo[data='baz.html']")
Floki.find("a b.foo")
The following examples can be implemented as well, but since depends on a more robust representation of the HTML tree, it should be implemented in the future:
Floki.find("a + b")
(matches b
followed by a
)Floki.find("a > b")
(matches b
that is right in the first level of children of a
)Floki.find("a:first-child") (matches
awhen
a` is the first child of its parent)This issue is related to #18
This refactor is needed in order to implement some features like search using pseudo-classes and update tree.
There is a prove of concept here, with all tests passing.
This refactor should not break APIs (as the name suggest). The only change that may occur is the Elixir version support.
html = """<html>
<head>
<title>Tag title - test</title>
<meta property='og:image' content='http://i.imgur.com/qQNJVAY.png'>
<meta property='twitter:image' content='http://i.imgur.com/O40KSbh.png'>
</head>
</html>
"""
Floki.find(html, "meta[property=og:image]")
[warn] Unknown token {:unknown, 1, ':'}. Ignoring.
[]
Hi. Basically I can't seem to use Floki.find/2 to get any properties with : in them (opengraph, twitter etc.) This bug seems to have popped up after v0.9.0. The above code works on 0.9.X, but breaks starting on 0.10.X
Thanks!
ref
floki/lib/floki/selector_parser.ex
Lines 62 to 63 in ba3b1e3
How about
Logger.debug("[floki] Unknown token #{inspect unknown}. Ignoring.")
?
Floki forces attributes' names lowercase. Failing test case:
defmodule FlokiTest do
use ExUnit.Case
test "preserves case of attributes names" do
xml = ~s(<root ID="1"/>)
[{"root", [{attrName, "1"}], []}] = Floki.find(xml, "root")
assert(attrName == "ID")
end
end
This is needed in order to respect the case-insensitive item of CSS3 selectors.
Via #37 it looks like handling unclosed tags is a goal for built-in HTML parser. But I was surprised that a tag name floki
was added to the parsed html tree here. Is this expected behavior?
iex> "<div>hello <h1>world</h1><script>alert('wat');</div>" |> Floki.parse
{"div", [],
["hello ", {"h1", [], ["world"]},
{"script", [], ["alert('wat');</div></floki>"]}]}
Actual results:
Floki.parse("<span>5</span> <span>=</span> <span>5</span>")
[{"span", [], ["5"]}, {"span", [], ["="]}, {"span", [], ["5"]}]
Expected results:
Floki.parse("<span>5</span> <span>=</span> <span>5</span>")
# or
Floki.parse("<span>5</span> <span>=</span> <span>5</span>", remove_text_nodes_with_whitespaces: false)
[{"span", [], ["5"]}, " ", {"span", [], ["="]}, " ", {"span", [], ["5"]}]
Note: this is really important in some cases. For example, please try parsing html generated from github markup (code samples).
Hi! Thanks so much for Floki!
I have a need to transform Floki nodes. For example, to turn all relative links into absolute links:
html = "<p><a href='/foo'>click!</a></p>"
Floki.transform(html, "a", fn({"a", [href: x], xs}) -> {"a", URI.merge("https://base.com", x), xs} end)
# => [{"p", [], [{"a", [{"href", "https://base.com/foo"}], ["click!"]}]}]
Instead of requiring the implementation of a recursive search, do you think there is space in the Floki API for a transform
function like above? I'd be up for cutting a PR myself. Just wondering if you have any thoughts.
I tried to update a project to the new 0.12.0
release, but ran into a bug. Here is a minimal reproduction of the error:
html = """
<html>
<body>
<div id="messageBox" class="legacyErrors"><div class="messageBox error"><h2 class="accessAid">Error Message</h2><p>There has been an error in your account.</p></div></div>
<div id="main" class="legacyErrors"><p>Separate Error Message</p></div>
</body>
</html>
"""
html |> Floki.parse |> Floki.find(".messageBox p") |> Floki.text
# => "There has been an error in your account.Separate Error Message"
The .messageBox p
selector should only return the first p
node, but it actually returns both of the p
nodes and the Floki.text
call ends up returning the text from both of those nodes.
Is it possible to update the dom elements?
I received an error when trying to parse the html of this page. Let me know if you need anything else.
** (FunctionClauseError) no function clause matching in Floki.Finder.traverse/4
lib/floki/finder.ex:33: Floki.Finder.traverse({:pi, "php echo date("Y"); "}, [" | ", {"a", [{"href", "http://crossatlanticebooks.com"}, {"target", "_blank"}], ["Cross Atlantic E-Books"]}, " , a division therein "], %Floki.Selector{attributes: [%Floki.AttributeSelector{attribute: "property", match_type: :equal, value: "og:image"}], classes: [], combinator: nil, id: nil, type: "meta"}, [])
lib/floki/finder.ex:40: Floki.Finder.traverse/4
lib/floki/finder.ex:40: Floki.Finder.traverse/4
lib/floki/finder.ex:44: Floki.Finder.traverse/4
lib/floki/finder.ex:40: Floki.Finder.traverse/4
lib/floki/finder.ex:23: Floki.Finder.find/2
(hue2) lib/hue2/tweet_info_2.ex:263: Hue2.TweetInfo2.get_source_data/1
(elixir) lib/enum.ex:1043: anonymous fn/3 in Enum.map/2
(elixir) lib/enum.ex:1387: Enum."-reduce/3-lists^foldl/2-0-"/3
(elixir) lib/enum.ex:1043: Enum.map/2
(hue2) lib/hue2/tweet_info_2.ex:98: Hue2.TweetInfo2.store/0
Created a new Elixir project (mix new floki_test
), added {:floki, "~> 0.4"}
to mix.exs
, successfully ran mix deps.get
, but when I run mix test
on this blank project, Floki fails to compile and the project outputs:
➜ mix test
==> mochiweb (compile)
Compiled src/reloader.erl
*** SNIP ***
Compiled src/mochifmt.erl
Compiled src/mochiweb_charref.erl
/usr/local/Cellar/erlang/18.0.2/lib/erlang/lib/parsetools-2.1/include/leexinc.hrl:47: attribute 'dialyzer' after function definitions
/usr/local/Cellar/erlang/18.0.2/lib/erlang/lib/parsetools-2.1/include/leexinc.hrl:118: attribute 'dialyzer' after function definitions
/usr/local/Cellar/erlang/18.0.2/lib/erlang/lib/parsetools-2.1/include/leexinc.hrl:194: attribute 'dialyzer' after function definitions
/usr/local/Cellar/erlang/18.0.2/lib/erlang/lib/parsetools-2.1/include/leexinc.hrl:247: attribute 'dialyzer' after function definitions
==> floki
could not compile dependency floki, mix compile failed. You can recompile this dependency with `mix deps.compile floki` or update it with `mix deps.update floki`
==> floki_test
** (Mix) Encountered compilation errors.
I'm using mix/elixir 1.0.5. Above steps succeed with floki 0.3.3.
Floki needs a better feedback of unknown tokens. The idea is to throw an error with the unknown token.
Hi,
Floki.find(html, "article")
returns nothing despite the fact the html does contain an article
tag. No other selectors find that tag too.
Thanks;
I want to be able to search for elements that are descendant of others without have to use
pipes.
Ideal interface:
Floki.find(html_tree, ".class-a .class-b")
It should be able to work with html tags and ids.
Floki is not finding elements in comments.
I have this src
I fetched with Hound
and I passed it to Floki.find(src, "span")
. Floki only returns the span
with the class='visible'
like this [{"span", [{"class", "visible"}], [" hello "]}]
. The commented out span is totally neglected.
Source:
<div class='parent'>
<div class='child'>
<span class='visible'> hello </span>
<!-- <span class='commented'> hi </span> -->
</div>
</div>
Is there a way to fetch commented out code? Or there's a temporary hack I can use?
You can reproduce this with:
Floki.find("<div class='parent'><div class='child'><span class='visible'> hello </span><!-- <span class='commented'> hi </span> --></div></div>", "span")
For this code:
<span>1</span> <a>2</a>
I run:
Floki.find(html, "span + a")
Expected results:
[{"a", [], ["2"]}]
Actual results:
[]
Note:
With this html:
<span>1</span> <a>2</a>
and this:
<span>1</span><a>2</a>
find method works as expected.
So problem is that Floki
interprets
(text node) as normal node. So next element (for Floki) after span
element is text node instead of a
element.
Note2:
As a workaround I tried:
Floki.find(html, "span + * + a") # any element between span and link
but I got:
** (MatchError) no match of right hand side value: " "
lib/floki/finder.ex:137: Floki.Finder.traverse_sibling/3
lib/floki/finder.ex:69: Floki.Finder.traverse/4
lib/floki/finder.ex:73: Floki.Finder.traverse/4
lib/floki/finder.ex:47: Floki.Finder.find_selectors/2
I get this trying to mix deps.compile floki
on my OS X machine:
/usr/local/Cellar/erlang/18.0.2/lib/erlang/lib/parsetools-2.1/include/leexinc.hrl:47: attribute 'dialyzer' after function definitions
/usr/local/Cellar/erlang/18.0.2/lib/erlang/lib/parsetools-2.1/include/leexinc.hrl:118: attribute 'dialyzer' after function definitions
/usr/local/Cellar/erlang/18.0.2/lib/erlang/lib/parsetools-2.1/include/leexinc.hrl:194: attribute 'dialyzer' after function definitions
/usr/local/Cellar/erlang/18.0.2/lib/erlang/lib/parsetools-2.1/include/leexinc.hrl:247: attribute 'dialyzer' after function definitions
==> floki
could not compile dependency floki, mix compile failed. You can recompile this dependency with `mix deps.compile floki` or update it with `mix deps.update floki`
Example:
<h2>Show user</h2>
<ul>
<li><strong>Name:</strong>Jim</li>
<li><strong>Age:</strong>25</li>
</ul>
<a href="/users">Back</a>
Floki.parse/1
retruns only first node {"h2", [], ["Show user"]}
It is, of course, :mochiweb_html.parse/1
. But can we make it better on Floki side?
It's really nice to be able to queries like:
table.some-table > tbody > tr:nth-child(1) > td:nth-child(2)
Support for floki would be awesome :)
The "attribute equals" selector is useful when one wants to search elements by the value of some attribute, like data attributes or "schema" properties.
Check the jQuery documentation for more details.
With @html
markup from test/floki_test.exs
:
<html>
<head>
<title>Test</title>
</head>
<body>
<div class="content">
<a href="http://google.com" class="js-google js-cool">Google</a>
<a href="http://elixir-lang.org" class="js-elixir js-cool">Elixir lang</a>
<a href="http://java.com" class="js-java">Java</a>
</div>
</body>
</html>
I am trying to search for an element with multiple classes (on the same element) with the selector .js-cool.js-elixir
but Floki is not finding them. However, if I add a space between the classes, as if the second was a descendent of the first, .js-cool .js-elixir
, it does return what I'm looking for but should not (as js-elixir
is not descendent of js-cool
).
Test demonstrating the error:
test "find elements with given multiple classes" do
class_selector = ".js-cool.js-elixir"
assert Floki.find(@html, class_selector) == [
{"a", [
{"href", "http://elixir-lang.org"},
{"class", "js-elixir js-cool"}],
["Elixir lang"]}
]
end
Hi @philss - thanks so much for Floki it's awesome...
I'm in a bit of a pickle with mochiweb_html conflicting with deps that use the mochiweb package.
I understand your reasoning only including the mochiweb_html parser rather than all of the other stuff with Floki, however when trying to build a release it causes the following error:
==> Failed to build release:
Duplicated modules:
mochiweb_html specified in mochiweb and mochiweb_html
mochiweb_charref specified in mochiweb and mochiweb_html
mochiutf8 specified in mochiweb and mochiweb_html
mochinum specified in mochiweb and mochiweb_html
Given that releases are borked and I'd have to fork your library (as would anyone building a release that includes mochiweb in the deps) would you consider this optimisation less desirable and go over to using full mochiweb?
Thanks again!!!
Hi guys, if I do Floki.find(html, "#wrapper")
for instance I very likely will get a lot of sub DOM elements, can't I convert it to HTML? I just saw Floki.text
, but I dont want just text.
Something like:
Floki.find(html, "#wrapper") |> Floki.raw_html
Result:
<div id="wrapper">... children DOM here ...</div>
iex(1)> "<ns:tag>xxx</ns:tag>" |> Floki.find("ns:tag")
Unknown token ':'. Ignoring.
[]
The jQuery can do as follows:
$("ns\\:tag")
I'm getting this error when parsing a XML file.
** (MatchError) no match of right hand side value: {"version", "1.0"}
Removing this line:
<?xml version="1.0" encoding="UTF-8"?>
From:
<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0">
<channel>
<title></title>
<link></link>
<description></description>
<language></language>
<copyright></copyright>
<lastBuildDate></lastBuildDate>
<docs></docs>
<image>
<url></url>
<title></title>
<link></link>
<width></width>
<height></height>
<description>
</description>
</image>
<item>
<title></title>
<link></link>
<description></description>
<category></category>
<pubDate></pubDate>
</item>
<item>
<title></title>
<link></link>
<description></description>
<category></category>
<pubDate></pubDate>
</item>
</channel>
</rss>
It seems to work just fine.
Any thought's on supporting this kind of file?
Hi,
I'd like to filter out comments, how do I do it? I tried
Floki.filter_out(doc, :comment)
But that doesn't work.
This is low priority.
The idea is to remove mochiweb code from this codebase.
Solutions are:
First off, I'm a big fan of this library - I really like the approach it's taking.
I'm using it to simplify my assertions in Phoenix Controller tests, and I find that a version of Floki.text/1 that returned text deeply would be super useful for me, letting me write tests that are agnostic to styling choices in the template. In the example in this method's documentation, I'm proposing something that would work like so:
iex> Floki.text("<div><span>something else</span>hello world</div>", deep: true)
"something elsehello world"
Thoughts?
I use Floki to parse XML. When I need to find "media:thumbnail" it returns me empty list.
Example:
iex(2)> Floki.find(response.body, "item") |> List.first
{"item", [],
[{"title", [], ["Test"]},
{"description", [], ["test"]},
{"media:thumbnail",
[{"url", "test-url"}], []},]}
iex(3)> Floki.find(Floki.find(response.body, "item") |> List.first, "media:thumbnail")
[]
Maybe I use Floki wrong? Thank you for help)
I get this error with floki 0.6.1
=> floki
could not compile dependency :floki, "mix compile" failed. You can recompile this dependency with "mix deps.compile floki", update it with "mix deps.update floki" or clean it with "mix deps.clean floki"
** (UndefinedFunctionError) undefined function: :leex.file/2 (module :leex is not available)
:leex.file('src/floki_selector_lexer.xrl', [scannerfile: 'src/floki_selector_lexer.erl', report: true])
(mix) lib/mix/compilers/erlang.ex:84: anonymous fn/3 in Mix.Compilers.Erlang.compile/3
(elixir) lib/enum.ex:1385: Enum."-reduce/3-lists^foldl/2-0-"/3
(mix) lib/mix/compilers/erlang.ex:83: Mix.Compilers.Erlang.compile/3
(elixir) lib/enum.ex:1043: anonymous fn/3 in Enum.map/2
(elixir) lib/enum.ex:1385: Enum."-reduce/3-lists^foldl/2-0-"/3
(elixir) lib/enum.ex:1043: Enum.map/2
(mix) lib/mix/tasks/compile.all.ex:19: anonymous fn/1 in Mix.Tasks.Compile.All.run/1
In my case, I'm attempting to use the pseudo-class :not
selector as "div:not(.invisible)#error"
pry(1)> Floki.find(html, "div:not(.invisible)#error")
iex(15)> [warn] Unknown token '('. Ignoring.
[warn] Unknown token ')'. Ignoring.
[warn] Pseudo-class "not" is not implemented. Ignoring.
[]
I would loooovvee to have support for :not
The idea here is to support child combinators in Floki.find/2
.
This issue is about the plus ("+") combinator.
Reference: http://www.w3.org/TR/css3-selectors/#adjacent-sibling-combinators
CSS example: https://developer.mozilla.org/en-US/docs/Web/CSS/Adjacent_sibling_selectors
# These call never return
Floki.find("", "a")
Floki.find("foobar", "a")
Floki.find("foobar<", "a")
# find/2 wants at least one tag
Floki.find("foobar<a>", "a") # => [{"a", [], []}]
# To be precise, it's ok if input matches /<[a-zA-Z0-9]/
Floki.find("foobar<a", "a") # => []
Today the search by id is using a generic lookup algo that searches inside all the nodes.
The search by id can be optimized by stopping the lookup in the first match of element.
We made html5ever and Floki work together through ex_html5ever thanks to @hansihe and the Rustler team! 👏
Using ex_html5ever
can be very useful those who need a more accurate parsing of the HTML. It solves the issues #50 and #75, for example.
Since ex_html5ever
relies on a Rust NIF, it requires the user to have Rust installed.
Also I ran some benchmarking that shows the ex_html5ever
version is faster from the average to big HTML files. ⚡️
UPDATE: the html5ever Elixir NIF was named to "html5ever" on Hex.pm, and the repository is now https://github.com/hansihe/html5ever_elixir
Test case:
iex> html = "<select><option value=\"foo\">Foo</option><option value=\"bar\">Bar</option><option value=\"barz\">Barz</option></select>"
iex> Floki.parse(html) |> Floki.find("select option + option")
** (ArgumentError) argument error
:erlang.hd([])
lib/floki/finder.ex:90: Floki.Finder.traverse_sibling/4
lib/floki/finder.ex:60: Floki.Finder.traverse/4
lib/floki/finder.ex:40: Floki.Finder.traverse/4
lib/floki/finder.ex:56: Floki.Finder.traverse/4
lib/floki/finder.ex:44: Floki.Finder.traverse/4
lib/floki/finder.ex:23: Floki.Finder.find/2
Floki version: 0.6.1
Suppose I want to filter out all divs that have a particular attribute. I can't see how to do that with filter_out
.
Consider this HTML. It has one div with a class
and one without:
html = "<body> <div class=\"has\">one</div> <div>two</div> </body>"
I want to find all the divs without a class
. I expected this to work, but it does not:
iex(44)> Floki.filter_out(html, "div[class]")
{"body", [], [{"div", [{"class", "has"}], ["one"]}, {"div", [], ["two"]}]}
More precisely, I want to work with a list produced by find
, so a closer approximation to my real code would be this:
iex(40)> Floki.find(html, "div") |> Floki.filter_out("div[class]")
[{"div", [{"class", "has"}], ["one"]}, {"div", [], ["two"]}]
Note that filtering in the div with class
works (using find
):
iex(42)> Floki.find(html, "div") |> Floki.find("div[class]")
[{"div", [{"class", "has"}], ["one"]}]
So does filtering out all the divs, regardless of their attributes:
iex(45)> Floki.find(html, "div") |> Floki.filter_out("div")
[]
Am I doing something wrong?
Add the "~" sibling combinator.
Specs: http://www.w3.org/TR/css3-selectors/#general-sibling-combinators
When I try to extract text from elements containing a comment-tag, the deep_text-extraction breaks. Instead of ignoring the comment, it throws a FunctionClauseError.
Here is a minimal example:
defmodule FlokiCase do
def fail_deep_text_with_comment do
"""
<a>foo</a>
<!--bar-->
<b>baz</b>
""" |> Floki.parse |> Floki.text |> IO.puts
end
end
I expected to have it returned "foobaz", but instead got this error output.
I hope this helps.
Unknown token '('. Ignoring.
Let's say I have the following code
<div id="msg"><span>sometext</span> Hello world</div>
How do I get the textnode 'Hello world'?
html
|> Floki.find("#msg")
|> Floki.WHAT_METHOD_CAN_I_CALL_TO_GET_"Hello world"?
Adding :floki to applications is giving this error:
Erlang/OTP 18 [erts-7.2] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]
=INFO REPORT==== 18-Jan-2016::15:35:13 ===
application: logger
exited: stopped
type: temporary
Example html:
<p>I want <a href="http://www.google.com">link to</a> <strong>article</strong></p>
When run through floki:
[{"p", [],
["I want ", {"a", [{"href", "http://www.google.com"}], ["link to"]},
{"strong", [], ["article"]}]}]}]
Maybe this is the intended result... which is perfectly fine. I was hoping to use floki for a weird use-case... basically run HTML through floki, parse through the floki response, removing tags/attributes that aren't allowed, and then re-create html. Essentially using floki to help me cleanse user inputted html to remove tags that aren't allowed and/or malicious javascript attributes (xss attacks).
Because of how I wanted to use it, the space between </a> and <strong> is actually important, otherwise "link to" and "article" will be touching each other.
Feel free to close the issue if it was intended.
I've tried to parse a minimal soap response
iex(3)> a = """
...(3)> <?xml version="1.0" encoding="utf-8"?>
...(3)> <soap:Envelope>
...(3)> <soap:Body>
...(3)> <SomeResponse>foo</SomeResponse>
...(3)> </soap:Body>
...(3)> </soap:Envelope>
...(3)> """
"<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<soap:Envelope>\n <soap:Body>\n <SomeResponse>foo</SomeResponse>\n </soap:Body>\n</soap:Envelope>\n"
iex(4)> Floki.find(a, "SomeResponse")
[]
and floki seems to not be able to parse anything
Is there a reason why Floki.attribute
returns a list?
Right now I find myself constantly needing to run hd
or List.first
when retrieving attribute's value:
~s(<meta property="og:url" name="twitter:url" content="https://example.com">)
|> Floki.parse()
|> Floki.find(~s(meta[property="og:url"]))
|> Floki.attribute("content")
|> hd()
Attribute name is always uniqe, so it should be possible to have hd/1
in Floki.attribute
itself?
According to the HTML5 CR, 8.1.2.3 Attributes:
There must never be two or more attributes on the same start tag whose names are an ASCII case-insensitive match for each other.
Floki needs a HTML parser built in, in order to remove the mochiweb dependency. This will enable more flexibility and better control of the parsing step.
The parser goals are:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.