Error adding feed / importing opml

Discussion:

Nick Grossman

2009-01-28 14:55:25 UTC

Hey there,

I ran into trouble today importing one melkjug feed into another -- is
that expected behavior? Here's what I did:

* From my jug: http://melkjug.org/nickyg/good_morning, I went to add
a feed, and entered http://melkjug.org/toppdzn/jug
* instead of adding the feed, I got a "no feeds found" error (see http://skitch.com/nickyg/bniw1/melkjug-good-morning)
* then, I tried exporting the OPML file from http://melkjug.org/toppdzn/jug
as a workaround, and importing it to my jug. When I did that, I got
an "error extracting feeds" message (see http://skitch.com/nickyg/bniin/melkjug-good-morning)

Thanks,
Nick

--
Nick Grossman
The Open Planning Project -- http://theopenplanningproject.org
nickyg-***@public.gmane.org
(917) 825-6590

Nick Grossman

2009-01-28 14:58:27 UTC

Permalink

Update: it doesn't seem to be just melkjug feeds; this feed (http://feeds.wired.com/wired/politics
) gave me the same result.

Nick

Post by Nick Grossman
Hey there,
I ran into trouble today importing one melkjug feed into another --
* From my jug: http://melkjug.org/nickyg/good_morning, I went to
add a feed, and entered http://melkjug.org/toppdzn/jug
* instead of adding the feed, I got a "no feeds found" error (see http://skitch.com/nickyg/bniw1/melkjug-good-morning)
* then, I tried exporting the OPML file from http://melkjug.org/toppdzn/jug
as a workaround, and importing it to my jug. When I did that, I
got an "error extracting feeds" message (see http://skitch.com/nickyg/bniin/melkjug-good-morning)
Thanks,
Nick
--
Nick Grossman
The Open Planning Project -- http://theopenplanningproject.org
(917) 825-6590

--
Nick Grossman
The Open Planning Project -- http://theopenplanningproject.org
nickyg-***@public.gmane.org
(917) 825-6590

Joshua Bronson

2009-01-28 15:43:37 UTC

Permalink

[also responding on the melkjug.org-feedback list]
Hey Nick,

Thanks for the heads up. As for the first problem, here's what's happening:
we are fetching the resource at http://feeds.wired.com/wired/politics,
getting back a text/xml document (which we identif as a feed), and passing
it to feedparser. However feedparser is choking on parsing the feed. From
the log on the live site:

2009-01-28 09:57:11,122 ERROR [melkjug.controllers.feed] Error locating
feeds in http://feeds.wired.com/wired/politics:
Traceback (most recent call last):
File
"/usr/local/melk/deploy/0.4.1/src/melkjug/melkjug/controllers/feed.py", line
134, in _find_feeds_in_url
ff, response, content = self._check_for_feed(url)
File
"/usr/local/melk/deploy/0.4.1/src/melkjug/melkjug/controllers/feed.py", line
170, in _check_for_feed
ff = feedparser.parse(content, url)
File
"/usr/local/melk/deploy/0.4.1/src/melk.feedparser/melk/feedparser/parser.py",
line 52, in parse
melkify(ff, url)
File
"/usr/local/melk/deploy/0.4.1/src/melk.feedparser/melk/feedparser/parser.py",
line 74, in melkify
e['melk_id'] = identify_entry(e, url)
File
"/usr/local/melk/deploy/0.4.1/src/melk.feedparser/melk/feedparser/parser.py",
line 93, in identify_entry
return melk_id(eid, normalize_feed_url(feed_url))
File "/usr/local/melk/deploy/0.4.1/src/melk.util/melk/util/hash.py", line
63, in melk_id
hash.update(iid)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 29-32:
ordinal not in range(128)

I will look into this further and get back to you.

As for the second issue, Luke can probably tell you more.

Josh

Post by Nick Grossman
Update: it doesn't seem to be just melkjug feeds; this feed (
http://feeds.wired.com/wired/politics) gave me the same result.
Nick
Hey there,
I ran into trouble today importing one melkjug feed into another -- is that
* From my jug: http://melkjug.org/nickyg/good_morning, I went to add a
feed, and entered http://melkjug.org/toppdzn/jug
* instead of adding the feed, I got a "no feeds found" error (see
http://skitch.com/nickyg/bniw1/melkjug-good-morning)
* then, I tried exporting the OPML file from
http://melkjug.org/toppdzn/jug as a workaround, and importing it to my
jug. When I did that, I got an "error extracting feeds" message (see
http://skitch.com/nickyg/bniin/melkjug-good-morning)
Thanks,
Nick
--
Nick Grossman
The Open Planning Project -- http://theopenplanningproject.org
(917) 825-6590

Luke Tucker

2009-01-28 15:49:38 UTC

Permalink

ie it doesn't work.

- Luke

Post by Joshua Bronson
[also responding on the melkjug.org-feedback list]
Hey Nick,
Thanks for the heads up. As for the first problem, here's what's
happening: we are fetching the resource at http://feeds.wired.com/wired/politics
, getting back a text/xml document (which we identif as a feed), and
passing it to feedparser. However feedparser is choking on parsing
2009-01-28 09:57:11,122 ERROR [melkjug.controllers.feed] Error
File "/usr/local/melk/deploy/0.4.1/src/melkjug/melkjug/controllers/
feed.py", line 134, in _find_feeds_in_url
ff, response, content = self._check_for_feed(url)
File "/usr/local/melk/deploy/0.4.1/src/melkjug/melkjug/controllers/
feed.py", line 170, in _check_for_feed
ff = feedparser.parse(content, url)
File "/usr/local/melk/deploy/0.4.1/src/melk.feedparser/melk/
feedparser/parser.py", line 52, in parse
melkify(ff, url)
File "/usr/local/melk/deploy/0.4.1/src/melk.feedparser/melk/
feedparser/parser.py", line 74, in melkify
e['melk_id'] = identify_entry(e, url)
File "/usr/local/melk/deploy/0.4.1/src/melk.feedparser/melk/
feedparser/parser.py", line 93, in identify_entry
return melk_id(eid, normalize_feed_url(feed_url))
File "/usr/local/melk/deploy/0.4.1/src/melk.util/melk/util/
hash.py", line 63, in melk_id
hash.update(iid)
UnicodeEncodeError: 'ascii' codec can't encode characters in
position 29-32: ordinal not in range(128)
I will look into this further and get back to you.
As for the second issue, Luke can probably tell you more.
Josh
On Wed, Jan 28, 2009 at 9:58 AM, Nick Grossman
Update: it doesn't seem to be just melkjug feeds; this feed (http://feeds.wired.com/wired/politics
) gave me the same result.
Nick

Luke Tucker

2009-01-28 16:28:33 UTC

Permalink

Turns out this wired feed is a somewhat interesting example.

The feed is not technically valid:
http://feedvalidator.org/check.cgi?url=http%3A%2F%2Ffeeds.wired.com%2Fwired%2Fpolitics

We can accommodate it pretty easily though. Filed a ticket for it
that we should be able to push through fairly quickly. http://trac.openplans.org/melkjug/ticket/338

- Luke

Post by Nick Grossman
Update: it doesn't seem to be just melkjug feeds; this feed (http://feeds.wired.com/wired/politics
) gave me the same result.
Nick

--
Nick Grossman
The Open Planning Project -- http://theopenplanningproject.org
(917) 825-6590

Luke Tucker

2009-01-28 15:29:45 UTC

Permalink

mm, yeah... these seem to be among the more egregious and obvious bugs
accreting on the live site.
/me takes a quick look and kicks self for not having decent ui tests.

- Luke

Nick Grossman

2009-01-28 15:42:35 UTC

Permalink

There are a few other weird things happening in my jug, too:

* as I mentioned in my previous email, certain feeds won't import
* some feeds, after importing, show up in the feed list with the URL
instead of the feed title -- this leads me to believe the feed may not
have fully loaded? (I would skitch but I already deleted that feed)
* now that I have a set of feeds that all seem legit, some of them
still won't produce articles in my stream. For example, see http://melkjug.org/nickyg/good_morning
now; I've added a wired news filter and turned it up, but melkjug
just keeps on chugging and won't refresh -- so far I haven't been able
to make those articles show up in my stream no matter what
* when melkjug gets chugging, other changes I make (e.g., deleting a
feed from my list) aren't actually saved on refresh.

That's a bunch of things, with no steps for reproduction... I can go
into more detail and reproduce if you want me to.

Nick

Post by Luke Tucker
mm, yeah... these seem to be among the more egregious and obvious
bugs accreting on the live site.
/me takes a quick look and kicks self for not having decent ui tests.
- Luke

--
Nick Grossman
The Open Planning Project -- http://theopenplanningproject.org
nickyg-***@public.gmane.org
(917) 825-6590

Luke Tucker

2009-01-28 15:49:23 UTC

Permalink

Sounds pretty hosed in total. Haven't been paying much personal
attention to it lately I have to admit.
I'll have a look today at what's going on...

The feeds showing up as URLs is a known behavior it does that when the
backend hasn't actually fetched
that feed yet.

- Luke

Post by Nick Grossman
* as I mentioned in my previous email, certain feeds won't import
* some feeds, after importing, show up in the feed list with the
URL instead of the feed title -- this leads me to believe the feed
may not have fully loaded? (I would skitch but I already deleted
that feed)
* now that I have a set of feeds that all seem legit, some of them
still won't produce articles in my stream. For example, see http://melkjug.org/nickyg/good_morning
now; I've added a wired news filter and turned it up, but melkjug
just keeps on chugging and won't refresh -- so far I haven't been
able to make those articles show up in my stream no matter what
* when melkjug gets chugging, other changes I make (e.g., deleting
a feed from my list) aren't actually saved on refresh.
That's a bunch of things, with no steps for reproduction... I can
go into more detail and reproduce if you want me to.
Nick

Post by Luke Tucker
mm, yeah... these seem to be among the more egregious and obvious
bugs accreting on the live site.
/me takes a quick look and kicks self for not having decent ui tests.
- Luke

Post by Nick Grossman
Hey there,
I ran into trouble today importing one melkjug feed into another
* From my jug: http://melkjug.org/nickyg/good_morning, I went to
add a feed, and entered http://melkjug.org/toppdzn/jug
* instead of adding the feed, I got a "no feeds found" error (see http://skitch.com/nickyg/bniw1/melkjug-good-morning)
* then, I tried exporting the OPML file from http://melkjug.org/toppdzn/jug
as a workaround, and importing it to my jug. When I did that, I
got an "error extracting feeds" message (see http://skitch.com/nickyg/bniin/melkjug-good-morning)
Thanks,
Nick
--
Nick Grossman
The Open Planning Project -- http://theopenplanningproject.org
(917) 825-6590

--
Nick Grossman
The Open Planning Project -- http://theopenplanningproject.org
(917) 825-6590

Luke Tucker

2009-01-28 15:55:53 UTC

Permalink

Post by Nick Grossman
* then, I tried exporting the OPML file from http://melkjug.org/toppdzn/jug
as a workaround, and importing it to my jug. When I did that, I
got an "error extracting feeds" message (see http://skitch.com/nickyg/bniin/melkjug-good-morning)

This should now be working on the live site. There was a fix laying
around in the trunk that never made it's way out somehow (?!) This
was related to change in the behavior of FormEncode.

http://trac.openplans.org/melkjug/ticket/312

- Luke

Nick Grossman

2009-01-28 16:01:23 UTC

Permalink

Post by Luke Tucker

Cool, thanks.

Post by Luke Tucker
- Luke

--
Nick Grossman
The Open Planning Project -- http://theopenplanningproject.org
nickyg-***@public.gmane.org
(917) 825-6590

Luke Tucker

2009-01-28 16:33:26 UTC

Permalink

My best guess at why this doesn't work is just that generating the
feed is too slow and melkjug gives up on itself.

- Luke

Randall Leeds

2009-01-28 18:55:01 UTC

Permalink

How do people feel about switching the feed associated with a jug to be a
planet-style aggregation of feeds in that jug rather than a call to
top_items?

Post by Nick Grossman
* From my jug: http://melkjug.org/nickyg/good_morning, I went to add a
feed, and entered http://melkjug.org/toppdzn/jug
* instead of adding the feed, I got a "no feeds found" error (see
http://skitch.com/nickyg/bniw1/melkjug-good-morning)<http://skitch.com/nickyg/bniw1/melkjug-good-morning%29>
My best guess at why this doesn't work is just that generating the feed is
too slow and melkjug gives up on itself.
- Luke

Luke Tucker

2009-01-28 19:29:31 UTC

Permalink

I'm thinking it would be better that way... faster for the default /
simpler to understand at first / push the fancy selection stuff off to
a different view. There is some stuff on the branch I've been playing
around with that should support that pretty directly.

- Luke

Post by Randall Leeds
How do people feel about switching the feed associated with a jug to
be a planet-style aggregation of feeds in that jug rather than a
call to top_items?

My best guess at why this doesn't work is just that generating the
feed is too slow and melkjug gives up on itself.
- Luke