Failed to include caffe_pb2

[tianyuandu]

Hi Yihui,

I tried your code and met some problems.

After make -j8 and make pycaffe, I tried to python3 train.py, but found something wrong with protobuf.
So I change the protobuf version but the problem was still not solved.

Here is the problem:
When I tried protobuf 3.0.0(b1,b2,b3,b4) or 3.1.0 , the error message is:

Failed to include caffe_pb2, things might go wrong!
Traceback (most recent call last):
  File "/home/dutianyuan/anaconda3/lib/python3.5/site-packages/google/protobuf/internal/python_message.py", line 1087, in MergeFromString
    if self._InternalParse(serialized, 0, length) != length:
  File "/home/dutianyuan/anaconda3/lib/python3.5/site-packages/google/protobuf/internal/python_message.py", line 1109, in InternalParse
    (tag_bytes, new_pos) = local_ReadTag(buffer, pos)
  File "/home/dutianyuan/anaconda3/lib/python3.5/site-packages/google/protobuf/internal/decoder.py", line 181, in ReadTag
    while six.indexbytes(buffer, pos) & 0x80:
TypeError: unsupported operand type(s) for &: 'str' and 'int'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train.py", line 19, in <module>
    from lib.net import Net, load_layer, caffe_test
  File "/mnt/lustre/dutianyuan/channel-pruning/lib/net.py", line 7, in <module>
    import caffe
  File "/mnt/lustre/dutianyuan/channel-pruning/caffe/python/caffe/__init__.py", line 4, in <module>
    from .proto.caffe_pb2 import TRAIN, TEST
  File "/mnt/lustre/dutianyuan/channel-pruning/caffe/python/caffe/proto/caffe_pb2.py", line 799, in <module>
    options=_descriptor._ParseOptions(descriptor_pb2.FieldOptions(), '\020\001')),
  File "/home/dutianyuan/anaconda3/lib/python3.5/site-packages/google/protobuf/descriptor.py", line 869, in _ParseOptions
    message.ParseFromString(string)
  File "/home/dutianyuan/anaconda3/lib/python3.5/site-packages/google/protobuf/message.py", line 185, in ParseFromString
    self.MergeFromString(serialized)
  File "/home/dutianyuan/anaconda3/lib/python3.5/site-packages/google/protobuf/internal/python_message.py", line 1093, in MergeFromString
    raise message_mod.DecodeError('Truncated message.')
google.protobuf.message.DecodeError: Truncated message.

and when I change protobuf to 3.2.0 / 3.3.0 / 3.4.0, the error message is

Traceback (most recent call last):
File "train.py", line 19, in <module>
  from lib.net import Net, load_layer, caffe_test 
File "/mnt/lustre/dutianyuan/channel-pruning/lib/net.py", line 7, in <module>
  import caffe
File "/mnt/lustre/dutianyuan/channel-pruning/caffe/python/caffe/__init__.py", line 4, in <module>
  from .proto.caffe_pb2 import TRAIN, TEST
File "/mnt/lustre/dutianyuan/channel-pruning/caffe/python/caffe/proto/caffe_pb2.py", line 17, in <module>
  serialized_pb='\n\x0b\x63\x61\x66\x66\x65.proto\x12\x05\x63\x61\x66\x66\x65\"\x1c\n\tBlobShape\x12\x0f\n\x03\x64im\x18\x01 
.....with a lot of \x......
  File "/home/dutianyuan/anaconda3/lib/python3.5/site-packages/google/protobuf/descriptor.py", line 824, in __new__
    return _message.default_pool.AddSerializedFile(serialized_pb)
TypeError: expected bytes, str found

When I first met this problem, my Python version is 3.4 and your default setting is with Python 3.5. So I install Python3.5 and Python3.6 by Anaconda, Python 3.6 by apt-get. No matter what version is, the problem was still not solved.

Hope you can show me the specific version of your coding environment!
Thanks!

[ethanhe42]

I encountered this before, however solved it with upgrade to protobuf 3.2
my version:

$ pip3 show protobuf
Name: protobuf
Version: 3.2.0
Summary: Protocol Buffers
Home-page: https://developers.google.com/protocol-buffers/
Author: protobuf@googlegroups.com
Author-email: protobuf@googlegroups.com
License: New BSD License
Location: /usr/local/lib/python3.5/dist-packages
Requires: setuptools, six

If you still suffer this problem, the alternative is install my ImageData layer to your own Caffe.
Replace the following parts:

/src/caffe/layers/image_data_layer.cpp
/include/caffe/layers/image_data_layer.hpp
/src/caffe/util/io.cpp
/include/caffe/util/io.hpp
src/caffe/data_transformer.cpp

and https://github.com/yihui-he/caffe-pro/blob/master/src/caffe/proto/caffe.proto#L783-L814

[ethanhe42]

whether these parts still works well in current caffe is not granted

[ethanhe42]

According to BVLC/caffe#2464, this problem is solved in latest Caffe version. Since I can't reproduce this problem, I've merge my Caffe to the latest BVLC/caffe. Please update and try remake.

[tianyuandu]

@yihui-he Thanks for your suggestion.
I've tried the same setting you mentioned with caffe in your repository but still didn't work.
Then I cloned the latest Caffe (before official Caffe solved this problem) and replaced the modified files, and built with protobuf 3.2 and python3.5, I can successfully include caffe_pb2 and run your code.

[LearnerInGithub]

@tianyuandu I clone the official caffe, and replaced the files you mentioned, but still face this problem. Does anything I'm missing? I installed the protobuf 3.2 and python3 is Python3.5.

[ethanhe42]

Can you built official caffe successfully in the first place? If not, please ask this question in Caffe.

[LearnerInGithub]

@yihui-he Yes, I can compile caffe successfully, problem will happy while I run the command:
python3 train.py -action c3 -caffe 0

[tianyuandu]

@LearnerInGithub
I'm not sure what your error message is. You can paste error message here.
After built official caffe, can you import caffe and import caffe.proto?
If import caffe worked and import caffe.proto failed, you can try to add these codes at the start of train.py after from __future__ import print_function:
import sys
sys.path.insert(0, "path/to/caffe/python")

Hope that can help you.

[LearnerInGithub]

@tianyuandu Thanks for your reply, when I import caffe, error will occurs:

Traceback (most recent call last):
File "", line 1, in
File "/channel-pruning/caffe/python/caffe/init.py", line 1, in
from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver, NCCL, Timer
File "/cephfs/person//channel-pruning/caffe/python/caffe/pycaffe.py", line 15, in
import caffe.io
File "/channel-pruning/caffe/python/caffe/io.py", line 2, in
import skimage.io
File "/usr/local/lib/python3.5/site-packages/skimage/io/init.py", line 15, in
reset_plugins()
File "/usr/local/lib/python3.5/site-packages/skimage/io/manage_plugins.py", line 93, in reset_plugins
_load_preferred_plugins()
File "/usr/local/lib/python3.5/site-packages/skimage/io/manage_plugins.py", line 73, in _load_preferred_plugins
_set_plugin(p_type, preferred_plugins['all'])
File "/usr/local/lib/python3.5/site-packages/skimage/io/manage_plugins.py", line 85, in _set_plugin
use_plugin(plugin, kind=plugin_type)
File "/usr/local/lib/python3.5/site-packages/skimage/io/manage_plugins.py", line 255, in use_plugin
_load(name)
File "/usr/local/lib/python3.5/site-packages/skimage/io/manage_plugins.py", line 299, in _load
fromlist=[modname])
File "/usr/local/lib/python3.5/site-packages/skimage/io/_plugins/matplotlib_plugin.py", line 3, in
import matplotlib.pyplot as plt
File "/usr/local/lib/python3.5/site-packages/matplotlib/pyplot.py", line 37, in
from matplotlib.figure import Figure, figaspect
File "/usr/local/lib/python3.5/site-packages/matplotlib/figure.py", line 40, in
from matplotlib.axes import Axes, SubplotBase, subplot_class_factory
File "/usr/local/lib/python3.5/site-packages/matplotlib/axes/init.py", line 4, in
from ._subplots import *
File "/usr/local/lib/python3.5/site-packages/matplotlib/axes/_subplots.py", line 10, in
from matplotlib.axes._axes import Axes
File "/usr/local/lib/python3.5/site-packages/matplotlib/axes/_axes.py", line 24, in
import matplotlib.dates as _ # <-registers a date unit converter
File "/usr/local/lib/python3.5/site-packages/matplotlib/dates.py", line 125, in
from dateutil.rrule import (rrule, MO, TU, WE, TH, FR, SA, SU, YEARLY,
File "/usr/local/lib/python3.5/site-packages/dateutil/rrule.py", line 55
raise ValueError, "Can't create weekday with n == 0"
^
SyntaxError: invalid syntax

[tianyuandu]

@LearnerInGithub
Actually I haven't meet this problem.
I just found a similar error in http://blog.csdn.net/yzf0011/article/details/73250107
You can try this:sudo pip3 install --upgrade python-util

[LearnerInGithub]

@tianyuandu I execute this command: sudo pip3 install --upgrade python-util:
Requirement already up-to-date: python-util in /usr/local/lib/python3.5/site-packages
Requirement already up-to-date: docopt in /usr/local/lib/python3.5/site-packages (from python-util)

However, the import caffe still have problem:

import caffe
Failed to include caffe_pb2, things might go wrong!
Traceback (most recent call last):
File "", line 1, in
File "/cephfs/person/mingliu/channel-pruning/caffe/python/caffe/init.py", line 4, in
from .proto.caffe_pb2 import TRAIN, TEST
File "/cephfs/person/mingliu/channel-pruning/caffe/python/caffe/proto/caffe_pb2.py", line 831, in
options=_descriptor._ParseOptions(descriptor_pb2.FieldOptions(), _b('\020\001')), file=DESCRIPTOR),
TypeError: init() got an unexpected keyword argument 'file'

[tianyuandu]

@LearnerInGithub Seem like caffe_pb2.py has some problem. Is your protobuf 3 install by pip3?
And when you are building caffe, the protobuf in system was chosen and maybe the version is mismatch with pip's protobuf.
I used cmake to build caffe so that I can see the exact version of protobuf, boost and cuda. You can also try this.

[LearnerInGithub]

@tianyuandu Sounds a good idea, I will try it. Thanks you!

[LearnerInGithub]

@tianyuandu @yihui-he protobuf problem fixed, because of the c++ version pb not matching the python version, reinstall them has fixed. Thanks two guys!

[yingning]

@LearnerInGithub ,you mentioned you reinstall protobuf and python ? which version of protobuf and python you used? I had the same problem and my protobuf is 3.5.0 ,python version is 2.7.13 of anaconda. what should I do to fix this problem?Thanks .

[zhongcl-thu]

@tianyuandu @LearnerInGithub Hello， I downloaded the latest version of caffe, installed anaconda, and installed both python 2.7 and 3.5. The protoc in usr/lib is version 2.5. If I don't install(conda install) protoc 3.2 (or later) in python and use protoc2.5 , then caffe can be made all. But once protoc3.2 (or later) is installed in python, i cannot make all caffe. I hope you can provide some advice, thank you.

[cateweb]

solved with sudo pip3 install sklearn-contrib-lightning

[LongruiDong]

@tianyuandu @LearnerInGithub Hello， I downloaded the latest version of caffe, installed anaconda, and installed both python 2.7 and 3.5. The protoc in usr/lib is version 2.5. If I don't install(conda install) protoc 3.2 (or later) in python and use protoc2.5 , then caffe can be made all. But once protoc3.2 (or later) is installed in python, i cannot make all caffe. I hope you can provide some advice, thank you.

i am facing same problem,have you solved the problem?

[SophieFeng]

I have solved this problem. I update protobuf from v2.5.0 to v3.2.0, it seems work! ps: I use python 3.5.0.